The city of Amsterdam has a lot to offer. Among this are huge amounts of open datasets produced by the municipality of Amsterdam. Approximately 5 years ago, an initiative started to make these datasets available for everyone through REST APIs. During a period of learning and exploration, more and more datasets became available. Until a point where refactoring became necessary.
A completely new product has been developed from scratch using Django and Django Rest Framework. REST APIs are now generated on-the-fly, based on so-called Amsterdam Schema definitions (json-schema based). The underlying database tables are filled with fresh data using ETL data pipelines built with Airflow and Databricks.
During this talk, the process of developing this open source product, the do’s and dont’s, the pitfalls and successes and the roadmap for the future will be described.
About Jan Murre
I am a long-time python enthusiast (started in 1999) and still quite fond of the language! My career started at the chemical department of the university where I fell in love with programming and IT technology. When the web started I immediately got hooked to web technology. After working a couple of years for several IT services companies, I started a web company with some former colleagues and there my python journey started. Initially with Zope and Plone, later on with Django, Flask and Pyramid. Currently I am a free-lancer for several years. I am still using python for the web, but data-engineering and cloud technology now has become heavily involved in the mix.
In my free time I like to read, to cook and to sing.
