Developing music apps, OneMusicAPI announcements and more.
Over the past months we’ve been asked for multiple new features, which we’ve been shipping to prospective (and now current) clients. We thought it was about time we cut a new API version, imported the docs to the website and made an announcement!
onemusicapi.com uses S3 to cache images that are loaded from upstream data sources and sent to clients. The bucket used for these images allowed anonymous upload, but this was never a security issue because files were only uploaded from EC2 instances that had a suitable EC2 IAM role.
We created an EC2 instance that didn’t have any EC2 IAM roles, and used it to add some images to the cache. After the process was complete (which took several days), we found that the added images were not accessible, and found that it was due to permissions errors from S3. Further investigation in the AWS console confirmed this, and also showed that the object ACLs weren’t accessible.
Posted on 07 June 2021.
We’ve recently been performing some memory optimisations on OneMusicAPI to improve the performance of the service.
One aspect we explored was the memory use of cached code. We use a lot of Scala in the codebase and were concerned that many classes were being generated, cached, and using up memory space.
So we broke out the tools and dived into some lesser-known Java tools to analyse how much memory is being consumed in the code cache.
Posted on 13 January 2021.
Currently, OneMusicAPI only has a REST-style HTTP interface. We're now releasing an SDK for PHP which makes performing OneMusicAPI queries and lookups much easier.
How is it easier? Well, before we just exposed a HTTP interface, to which you could send query parameters and the like and receive JSON or images in response. Using the SDK you don't need to know worry about anything at the HTTP level and can just concentrate on using the service in your language of choice.
Posted on 14 October 2020.
onemusicapi.com allows searching for releases by passing in track
positions and durations. The search allows the track durations to be off
by a couple of seconds, (TODO figure out the reasons for variation), so
we need a fast method for performing fuzzy search on the track
durations.
Another requirement that makes things more complicated is that not all the input tracks need to match; a subset of the tracks matching (the actual fraction of tracks that need to match depended on other factors) need to be enough for the medium to match. However, this won’t be covered in detail here.
Posted on 08 October 2020.
onemusicapi.com runs on AWS Elastic Beanstalk with a Tomcat 8.5 platform, using t3.micro instances. t3.micro instances have 1GB of RAM, and Elastic Beanstalk’s monitoring puts instances in a Warning state if memory use exceeds 90% of available memory. We found that about a day or two after an EC2 instance was brought online, it would go into Warning state, due to memory use, even if the service wasn’t actually affected.
TL;DR: We found Tomcat was using more memory than expected, and the following helped:
-Xmx
We've just updated our pricing page to reflect our new pricing model.
Broadly speaking: the plans have been simplified. All plans now allow image download. We've removed the old unlimited plan, and replaced it with two higher tiers, both finite in the number of requests they offer. The free evaluation tier remains.
Posted on 08 January 2018.
While preparing a data extract I recently discovered a set of Discogs data which I've begun calling "special entities". These are artists or labels which aren't actually artists or labels, but are intended to work as a meta-entity for grouping or classification purposes.
Why is it worth calling these out? Well, depending on the way you are using the Discogs database or API, it may cause performance issues if you link to these artists or labels with little benefit; some are huge in the number of entities they link to.
Posted on 17 May 2016.
A new release of OneMusicAPI is in production, and we've rolled in a whole bunch of changes and useful new features to the API. As well as a few performance improvements, we've added support for returning the record label for a release, the composers for the tracks on a release and also whether the release is a compilation or not.
Posted on 25 April 2016.
We get a lot of visitors to the OneMusicAPI blog looking for tips working with Discogs' API. Our tips on better Discogs searching article is one of the most visited on the site.
Discogs is a great resource and contains a wealth of musical metadata. However, they are known for making breaking changes to their API, and they've recently announced a new one: going forward, all queries to Discogs should be made over HTTPS.
As a result, app developers need to update their own apps so they work with the new changes. That can be a pain for certain types of apps, because the cost of getting the new versions of the apps to their users is high (not just financial cost; I'm talking about the effort and time it takes to announce new versions etc). This is why OneMusicAPI exists of course!
Posted on 03 February 2016.
We've recently been doing some work on reducing the time it takes to perform an import from MusicBrainz, Discogs and our other sources into ADB.
Before, the import would take a long time. The Discogs import on its own, for example, would take over 24 hours! It was easy to see why: the import process was highly network-bound. Despite optimising the way we write to SimpleDB and CloudSearch, much of the time during the import was waiting for uploads to be made to these services.
Posted on 20 January 2016.
A letter in the mailbag this week:
Posted on 06 January 2016.So one thing I noticed when performing some test queries from the browser is that at first I got a couple of "No rule matched" errors until I decided to include the "20150623" part (a date?) in the query URL.
I didn't see a reference to this string in the intro docs. Is this an API version identifier? If so, where are allowable values documented?
I thought it worth announcing: we are now bronze supporters of the MusicBrainz project!
Posted on 26 November 2015.
I received a question by email a couple of days ago:
I'd like to understand an aspect of the returned data if I may, before seriously considering purchase. I'm interested in the 'score' value returned - what does mean? Considering it is returned as a '1' in both the track and artist documentation examples on your website, I suspect it is something to do with the quality of the data returned from the query, nothing to do with review score?Posted on 11 November 2015.
We've been getting more and more requests for data dumps of our music metadata database here at OneMusicAPI. Regular readers know OneMusicAPI is an aggregated database, built by combining the major free online databases: MusicBrainz, Discogs, Acoustid, Wikipedia, the Cover Art Archive and FreeDB.
Posted on 28 October 2015.
When it comes to images, the most common use of OneMusicAPI is in finding front covers for CDs, vinyl and other media, and artist images. But some music collectors also want to collect the back covers, insert images and other otherwise-physical media delivered with a given musical release.
Posted on 14 October 2015.
OneMusicAPI supports fingerprint based queries for both release and CD information. But how do you
generate a fingerprint to send to the service? One option is to use fpcalc, part of the
Chromaprint project.
I thought it was worth drawing out the difference between the inc and
mustInc parameters. These parameters are used to define metadata that
should be included in responses for releases (albums, singles, EPs, compilations etc).
The most common use is when requesting to include image metadata; URLs and resolutions.
It's time to make the first changes to our pricing plans.
I know what you're thinking: price changes, that means price increases, right? Well, in one case, yes. But in the other cases we've kept prices static and we've also introduced a new low usage plan to cover the gap that used to exist between the pre-pay and unlimited options.
Posted on 19 August 2015.
I'm pleased to announce our major new release: track metadata!
I always felt that once we added tracks to the existing release and artist APIs we'd have a much more complete API. Well, now we have, and we continue to offer our CD and musical image APIs too.
Posted on 10 August 2015.
Discogs is an online music database with enormous breadth and depth. At the time of writing this blog post, musical releases in the database number a whopping 6,121,612. For each release, metadata are provided about the release; and amongst this data are album covers. This makes Discogs one of the primary sources for album covers on the Web.
Posted on 10 July 2015.
FreeDB is a database of audio CDs, containing the CD's title, overall artist, year of release and details about each individual track. It is queryable in a few ways, most commonly using a CD's TOC (table of contents) which gleans a set of matching CDs with their FreeDB IDs. This ID can then be used to lookup the individual CD information.
Posted on 12 June 2015.
We've just uploaded a new version of our pricing page confirming something we often offer in emails: annual plans.
Posted on 11 May 2015.
A couple of days ago we made a new release available at api.onemusicapi.com. This
included support for Discogs' new approach to image serving, improved performance and also
better accuracy and matching coverage for short CDs in the /disc endpoint.
MusicBrainz is a fantastic, free (libre and gratis) online music database. A wealth of metadata is stored there; at the time of writing it covers 1.4m releases and almost a million artists. Perhaps more impressive than the breadth is the depth; it contains information about record labels, who performed on which track and much, much more.
Posted on 27 February 2015.
We're proud to announce a new version of OneMusicAPI... the biggest addition being artist metadata searches! We've also made several improvements to release queries; allowing query by Discogs release ID and also allowing "unknown" positions for tracks in fingerprint queries.
The changes in the API for this release means we've introduced a new API version, 20140925,
which we encourage all clients to migrate to. But as ever, we won't stop older versions whilst
paying customers are using them!
Acoustid is a great audio fingerprinting service. Open, gratis for non commercial users and with an enormous database of audio fingerprints, provided by a large userbase. This makes it easy for app developers to get up and running, coding against the Acoustid web service and resulting in a useful means by which music can be identified.
Inaccurate data does exist though, so read on to see how to use the Acoustid API to filter out errant entries....
Posted on 14 January 2015.
FreeDB remains the largest free source of musical release metadata. In 2012 the number of releases numbered over three million, more than both Discogs and MusicBrainz.
Unfortunately, FreeDB's metadata is limited, both in terms of the breadth of the metadata and its form. Textual metadata comprises just album and track titles, artists, genres and release year. That's a far cry from the rich data provided by the likes of MusicBrainz. The quality also seems generally lower than other online music databases.
Posted on 03 December 2014.
In the spirit of 'publish what you discover' I decided to make a record of a regular expression I recently used for analysing incoming MP4 atom fields. Specifically, breaking down the names of these fields to do something useful with the contents.
Posted on 05 November 2014.
SPARQL is a neat language for querying datasets stored in the RDF format, an example of which is DBpedia, a structured form of Wikipedia. The sending of a query to DBpedia from Java code is trivial, but the exact format of the SPARQL query needs to be considered in case it contains SPARQL-reserved characters or syntax. I couldn't find an easy SPARQL escaper, so I wrote one myself.
Posted on 08 October 2014.
Amongst the well-known sources of music metadata online, Wikipedia is often forgotten. MusicBrainz and Discogs are the free sources that seem to get most of the attention, but it is little discussed that Wikipedia contains several hundred thousand metadata records, mostly for mainstream releases.
Not only that, but the metadata provided by Wikipedia includes album artwork, which means it's a possible source for people interested in finding and downloading cover art.
Posted on 10 September 2014.
It's been all change for clients of the Discogs API this year. Since early in the New Year, when OAuth authentication was first required for image downloads, through to now, a stream of changes to how clients interact with the API appear to have worrying implications for app developers.
Posted on 13 August 2014.
Many software developers were flummoxed at the start of the year when it was announced that Discogs would start requiring OAuth authentication for image downloads. At the same time, a new limit of 1000 image downloads per day, per application, was imposed.
Posted on 16 July 2014.
"Powered by" logos are small graphics you can place on your website or product to communicate a little more about how your product works. For example, you can add "powered by Paypal" logos to show that Paypal perform credit card processing for your product. This helps build trust and credibility with your customers.
Posted on 18 June 2014.
Another week, another new release. This release includes a new API version to clean up of our returned JSON and also some accuracy improvements.
Posted on 28 May 2014.
Last week we announced the latest version of OneMusicAPI with an image endpoint, so images can be downloaded directly from OneMusicAPI.
This is useful because it provides greater stability; we won't stop or change APIs used by paying customers, so you know you will always be able to access the images. Furthermore it allows us to offer a cache of images within Discogs, avoiding their current 1000/day download limit.
With these cool new features there was something missing... a pricing plan to offer them as part of!
Posted on 14 May 2014.This new release contains a new feature: an image endpoint. What's an image endpoint? It's a way of downloading a selection of images directly from OneMusicAPI.
So what's the big deal? Couldn't you already just download the artwork from the existing URL served up by OneMusicAPI on some-other-domain? Yep, you could, but with caveats.
Posted on 06 May 2014.
We've just committed a new version of OneMusicAPI - 20140401. There are no API changes, as
such, so you should see all the benefits without changing your code. In this release we've concentrated
on accuracy and coverage improvements: bringing back the correct results, and more of them!
I know, from first hand experience, how difficult it is to launch and bootstrap a software project. How difficult it is to estimate how long development will take and how daunting it is to learn how to market and promote your project. Then there's managing all of those development risks: making sure different platforms and configurations are supported, proving that certain solutions can work, and keep working.
Posted on 12 March 2014.
I'm pleased to announce we have completed and launched the latest version of OneMusicAPI! This release's main new feature is the inclusion of the MusicBrainz database in the Aggregated Database (ADB).
Posted on 22 January 2014.
Since launching the Aggregated Database (ADB) OneMusicAPI has been working away a lot more efficiently. Over the past couple of weeks OneMusicAPI has had to scale to two EC2 micro instances five times, a massive improvement on before when we were constantly running between two and four instances to keep up with demand.
Posted on 11 December 2013.
One thing I've learnt building bliss is that my original desire for a tool to fix music library consistency was shared by others. As I have moved bliss's basic metadata lookup code into the cloud in the guise of OneMusicAPI, I'm now thinking about how other aspects of bliss can be integrated into OneMusicAPI, and how they can help OneMusicAPI's users.
bliss's focus on intra-library consolidation and consistency can be seen as a focus on personalisation. For example, "I want all my albums to fit into one of these genres" or "I want all track numbers reported to be left padded, if required, to three characters". If all of bliss's users want preferential control over their music libraries, it stands to reason that OneMusicAPI's users (or their users!) do too.
So the far off vision for OneMusicAPI may be MYMusicAPI (and yes, I have purchased that domain name!).
Posted on 13 November 2013.Good news! This is the first release of OneMusicAPI using its Aggregated Database (ADB). This redesign of OneMusicAPI is designed to improve performance and availability by avoiding costly and fault prone calls into other APIs.
Posted on 30 October 2013.
A regular expression is a textual expression that can be used to match other strings, or portions of strings. Regular expressions ("regexes" as they are also commonly known) are very powerful and are capable of describing and matching strings in multiple ways.
I've written before how you should strip disc number artifacts from release titles before querying. So, given a release title, what's the best way of recognising these disc number artifacts ("disc A", "disk II" etc) and removing them from release titles before querying? Using regexes is one way. Within OneMusicAPI and bliss I have been using and refining a regular expression to strip disc number artifacts for some years.
Posted on 16 October 2013.
This week's blog post is a guest post from the creator of the AudioGlu music server, Ian Smart. AudioGlu is an interesting use case; it's a piece of hardware that has adopted OneMusicAPI to install metadata and cover art when ripping CDs.
... and apologies for this post's image, but given Ian's feisty approach to building a music server I couldn't resist the chance to post an equally feisty chipmunk. Take it away Ian...
Posted on 02 October 2013.
Programmers of music apps, or any products that need to access information about music, need to look up music metadata somehow. Normally this is achieved by calling an online database via its API. The programmer develops code to send a message to the API asking for music information by sending some identification of the entity (e.g. to find out information about a release, the release name is sent).
Posted on 17 September 2013.
One for Java programmers who work with music files; MP4 files to be precise. MP4 is a popular file format; it's the default choice used by iTunes, so it's extensively used. MP4 is a container format which means it is capable of containing multiple data streams (audio, video etc) as well as metadata about those streams. It's an aspect of that metadata that's the subject of this blog post.
Posted on 04 September 2013.
Any software, hardware or website that needs to show music metadata needs access to a music metadata lookup service. There are free data sources for such information but they are not as large as the commercial databases. By combining the databases it is possible to improve the coverage and extract even more metadata; in this blog post I introduce a technique for this called "lookup chaining".
Posted on 28 August 2013.
It's been almost a month since I announced I had started work on OneMusicAPI's Aggregated Database (ADB is the rather unoriginal codename used internally... better suggestions welcome!). So, I thought: a good time to update you with work done so far, how the ADB is shaping up, and lessons learned!
Posted on 07 August 2013.
While OneMusicAPI is, and will continue to be, an aggregated music metadata API which represents multiple aggregated databases, there are multiple ways of achieving this. The current way owes much to OneMusicAPI's ancestory of being part of bliss, but I am now starting work on a more scalable approach which should improve availability and lower query latency.
Posted on 10 July 2013.
I've been using the JAudiotagger library for a number of years now, to tag music files in bliss. It's stable and well supported, but while profiling the other day I noticed a potential performance improvement.
Posted on 26 June 2013.
A while ago I wrote an article on the elsten software blog about best practices for searching Discogs. Since then there have been a few updates to the Discogs API, so I thought I'd repost the article with updated information and tips.
Posted on 12 June 2013.After a few months of development, the new release of OneMusicAPI is now released! There's a whole bunch of additional features in the new version including general release metadata, disc queries and vastly improved coverage for Acoustid fingerprint and MusicBrainz disc ID queries. OneMusicAPI also now supports FreeDB queries for discs!
Posted on 29 May 2013.Following some feedback by new users and customers of OneMusicAPI, I've written up some extra documentation.
Posted on 19 February 2013.The first release of OneMusicAPI has just been made! Earlier on I promoted the current beta version, 20120130, into production. It's great to finally release OneMusicAPI to general availability!
Posted on 30 January 2013.Whilst OneMusicAPI moves from beta to release (hopefully next week!) I thought I'd show off the latest non-technical developments... a new logo!
Posted on 25 January 2013.Just a quick note to announce the very first beta for OneMusicAPI. I've uploaded the first beta version which provides cover art given various inputs.
Posted on 28 November 2012.