We’re always looking for ways to improve the experience of our users, and we know that delivering our data as quickly as possible is an important part of that. We’re committed to lowering our delivery latency while maintaining the quality of our data in the process.
Below is a summary of the OpenCorporate’s data journey, as well as an outline of all our key issues and challenges involved in lowering latency.
Understanding the data journey
- 1. Getting the data (data fetching)
- How often? Depending on the type, we acquire the data hourly, daily, weekly, twice a week or quarterly.
- How long? Daily acquisitions can take as little as 5 minutes to process due to the smaller volumes of data involved. Larger downloads might take up to a week or two to process.
- 2. Preparing the data (data processing)
- How often? This happens right after we fetch the data, as we get it ready for ingestion into our system.
- How long? It’s part of the fetching process, so no extra time is needed.
- 3. Adding new data (data ingestion)
- How often? Our system checks every 5 minutes for new data to add to our main dataset.
- How long? Some data takes over 24 hours to add, but most are done in less than a day. There may also be some delays in starting the ingestion process as we can only have a limited number of jurisdictions ingesting at any one time.
- 4. Getting data ready (data extraction)
- How often? We go through all our data sources every four days. If there are any issues, it might take up to a week.
- How long? It can be as quick as a minute or up to 12 hours, depending on the size of the data.
- 5. Sending data (data delivery)
- How often? This depends on the specific needs and schedules of our customers.
- How long? Usually, it takes between 10 minutes to an hour.
Key points about our data process
Data retrieval timing: The date we retrieve the data might not be the same as when it’s available in our systems.
Data processing varies by location: Each jurisdiction has its own way of fetching and processing data, so the time it takes can vary.
Possible technical challenges: Sometimes we face technical issues that can cause delays, like slow processing or changes in data formats by registries. Typically these are infrastructure problems due to our systems being overloaded. A common example is when lots of different services are accessing our main SQL database at the same time, causing various processes to fail as requests to the database begin to time out.
For bulk deliveries, we aim to be no more than a month behind the registry.
We’re committed to improving these processes and minimising any delays in providing you with the most up-to-date data. If you have any concerns or would like to know more, please feel free to reach out to [email protected].