With Conserv Cloud, you can bring in data from your existing data loggers! Here are a few things to note when importing.
The Import Process
We've tried to give you a completely hassle-free import process, but sometimes issues come up.
To get your import working, it's important to understand the overall import process and to identify which step is causing a problem.
The Conserv import proces has five steps.
- Choosing a sensor type
- Selecting file(s) to be imported
- Mapping your data fields to Conserv
- Importing the data
- Mapping your import to new/existing sensors
This article will walk through these five steps to help you identify the source of your problem.
Before Getting Started - Prepare Your Files for Import
If you're using PEM2 or HOBO data loggers, we have specific articles walking you through getting your files setup for import.
📝 Import Data from PEM2 and eClimateNotebook
📝 Import Data from HOBO UX & HOBOware
📝 Import Data from HOBO MX Bluetooth and HOBOlink
Tip:
If you're using any other type of logger, the best way to ensure a successful upload is to use our Import Template.
You don't have to use the import template, but we strongly recommend it, as it is possible to get a large host of import problems related to time zones and duplicates.
A very important note about time zones
In some cases, data exported by loggers may include a timezone specified in the header but lack a timestamp in the date/time fields. When this happens, the import process will set the time zone to Universal Time Coordinated (UTC), which sometimes coincides with London time depending on the time of the year (due to British Daylight Savings). This can cause issues with analytics, as the time data may appear off by several hours. Here are the two main issues related to time stamps which may cause trouble when importing a file:
- Problem 1: Your date format is not appropriate. If your file has a date that looks like this or similar: 03/19/23 05:47, the software will not recognise it very well and it might think that it is UTC time, making your time stamp come out several hours behind your real reading time. To fix this first, you need to change your date format for those cells to Date Time so that your timestamp will look like this instead: 03/19/2023 05:47:00 or it can look like this and still work: 2023-03-19-11T05:47:00Z.
- Problem 2: If your timestamp looks like the second option, "2023-03-19T05:47:00Z", the file will be imported in UTC meaning that you analytics will appear to be 5 or 6 hours off (depending on whether the Americas are on daylight savings or not). This is because the software will think that reading is for March 19th 2023, at 5:47 UTC time (roughly London time).
To correctly import the data for your own time zone, the date/time value should include the time zone offset.
Tip: The type of timestamp your particular .csv file will depend on your original software and brand of non-Conserv logger. Some brands will state the time zone in the stamp while some will not. For that reason, and to avoid all kinds of confusion with time zone differences, we strongly recommend that you create a separate column in your file with a UTC-corrected timestamp that you can then use to map your data during the Conserv import procedure. This will ensure that when you look at your data in analytics, you see the values at the expected times.
Note: Data is presented to you in your specified time zone within the app.
To help you with this conversion, we have a sample Google Sheets file that uses a formula to convert a date/time that doesn't include the time zone offset into UTC. We have also added a second tab in that file that gives you some examples on what the formula looks like depending on what time zone you may be in.
The "+ (H/24)" in the formula below is the indicator that the time you have in your original file is from a time zone that is H hours behind UTC. In other words, UTC is H hours ahead. When it was 5:47am for you, it was 10:47am in UTC.
This is the formula we use to turn your original time stamp into a UTC time where B2 is the cell where your time stamp is and H is the number of hours difference:
=TEXT(B2 + (H/24),"yyyy-mm-ddThh:mm:ss")&"Z"
If the formula above gives you an error, make sure you have solved Problem 1 first as the formula will not work if the original date format is not correct.
Please make sure to set the time zone offset in the formula to the correct value for your time zone!
For example, for EDT, you would set the value to "+ (4/24)" between about March and November, but "+ (5/24)" between November and March (because of daylight savings changes).
Before you upload your final csv, make sure that your UTC column results make sense. If you are in the Americas, UTC times should always be ahead of your original time stamp.
Noon for you in the East Coast will be either 4pm or 5pm UTC depending on whether Daylight savings is on or off at the time. If you are in the West coast or somewhere in between, you will have to adjust the number of hours difference.
Check the formula you use for + or - (H/24) to make the UTC time zone reflects correctly. If you are in Europe east of London, or anywhere else where you are normally ahead of UTC time, you will need to adjust the formula to - instead of +.
Nathan's Tips for Cleaning Up Your Data Logger File
1. Try to avoid empty columns.
Was that a bit confusing? Here's a little video to help you out.
Step One - Choosing a sensor type
If you're using data from HOBO sensors, select "Onset".
If you're using data from PEM2s or eClimateNotebook select "IPI".
For all other csv files select the "Other Loggers" option. We've designed our import to work well with any csv files, assuming you've prepared the files correctly in Step One.
💡 Good news. This step is very straightforward.
💻 Here's what Step One "Choosing a Sensor Type" looks like in Conserv
Step Two - Selecting file(s) to be imported
Users on the free version of Conserv can import as much data as they want, but the import is limited to one file at a time.
💻 Here's what Step Two "Selecting file(s) to be imported" looks like in Conserv
💡 Users on the Conserv Essentials subscription can use multi-file import to import up to 20 files at once.
⚠️ If you're uploading multiple files, be sure that your formatting is the same across all of the files, otherwise the import will not proceed
KNOWN ISSUES
- Conserv has a limit of 20 files at once for data import. If you have more files than that, you'll have to do multiple imports.
Step Three - Mapping Your Data Fields to Conserv
The fields on the right come from the data you're importing. The fields on the left are where you're data will land in Conserv.
The software tries to automatically map your fields to Conserv. You can manually change a mapping by dragging and dropping the fields.
📺 Here's a quick video showing the drag and drop feature
KNOWN ISSUES
- Sometimes the mappings are completely wrong. Unusual file structures or languages can cause issues. The easiest way to ensure success is to move the data into the template that Conserv provides.
Step Four - Importing the data
Once you click "Next Step" on the mapping screen, your import process will begin.
💻 Here's what Step Four "Importing the data" looks like in Conserv.
KNOWN ISSUES
- Conserv has a limit of 50mb for data import files. Depending on the specific file format, this can be millions of readings and should be enough for most use cases. If you are trying to import more data than that, you can split the file into multiple parts.
- Disable pop up blockers or whitelist the Conserv.io domain. In some cases, ad blocker or popup blockers have blocked people from importing. Handling specific ad or popup blocker products is beyond the scope of this article. Please consult the instructions for your browser or security products for specific details on how to whitelist Conserv.io.
Step Five - Mapping your import to new/existing sensors
To complete the import you'll need to associate your imported data with a new or existing sensor in Conserv.
💻 Here's what Step Five "Mapping your import to new/existing sensors" looks like in Conserv
Deleting an import file
Sometimes you need to back out an import after it's completed, and that's something we can do!
From your list of imports, find the import you want to delete and click the "..." on the right side of the screen. From there you'll be presented the option to "delete import?
Create a Support Ticket
Hopefully this article solves your import problem, but if you're having import issues and feel like you need more support just let us know by creating a ticket here.