In libddog we provide a library API that maps onto the Datadog (service) API and, indirectly, onto the Datadog UI that users see and interact with. We do not control the Datadog API, and our understanding of it will always be imperfect. It will always be limited to specific cases we have seen of dashboards and widgets exported to JSON that we can learn from. This information feeds into the assumptions we make about Datadog's internal data architecture, and our design decisions will often fall out of that. The trick is not to be too closely tied to Datadog's representation of metrics and widgets - giving us some wiggle room to choose the API we think is best for users, and to allow us to evolve things internally in libddog without making breaking changes - but without going as far introducing whole new abstractions that don't exist in Datadog and which could become very difficult to sustain over time.
Our goal is provide a stable and understandable API which keeps working, even when the Datadog API may slowly evolve over time. Whenever possible we want to avoid breaking changes and keep user code working.
If we deem that breaking changes are necessary this must be reflected in the version by bumping the major version.
libddog is still a young project and this is reflected in the current version being in the 0.0.x series. At some point we will bump to 1.0.0 to reflect that it has become quite a mature project.
When making changes to the public API this must be reflected in the version:
- A bug fix or a small incremental feature (like a new kwarg added to an existing method) warrants a patch version bump:
1.2.3 -> 1.2.4. - A significant new feature added (like a new class or method) warrants a minor version bump:
1.2.3 -> 1.3.0. - A breaking change (a public API removed or changed in a backwards incompatible way) warrants a major version bump:
1.2.3 -> 2.0.0. Instructions must also be provided in the CHANGELOG on what code changes users need to make in order to upgrade.
As much as possible we should aim to avoid breaking changes by making additive changes only, eg. add a new method which supercedes an existing method (and mark the existing method deprecated) instead of changing an existing method.
bin- command line tools, ie.ddogci- executables that are needed for CIdocs- documentationdocs/skel- an example skeleton project for userslibddog- the libddog library itselflibtests- library code required by our teststestdata- code that consumes libddog as a library, used by our integration teststests_integ- the integration teststests_unit- the unit tests
When libddog is installed using pip only bin and libddog will be installed.
We have identified the following risks that we are attempting to mitigate with our QA process:
- We've misunderstood how something works in Datadog or just implemented it wrong.
- Code inside libddog is inconsistent/broken and can turn into latent TypeErrors, ValueErrors, NameErrors, ImportErrors and the like.
- Code inside libddog is not very readable, is brittle and painful to maintain.
- Some commands in
ddogare broken, or don't work given certain inputs. - The way we package and release libddog is broken and leads to released versions being broken or unusable.
Here is how our testing efforts map onto these risks:
Our integration tests target #1 and aim to fully (or close to fully) exercise the libddog API against the Datadog API. This is our best defense against changes in the Datadog API breaking libddog over time. Our tests consist of dashboards modeled using libddog and pushed to Datadog. We also verify that the update succeeded by fetching the dashboard from Datadog and comparing the JSON document we sent to the one Datadog echoes back to us. Modulo of a few unique identifiers, timestamps and other fields the client doesn't (or doesn't want to) control they should be identical.
As usual with integration tests, they are slow and inconvenient to run. They require a Datadog account and valid credentials, so they cannot be run in CI. The dashboards they create should also be checked visually in Datadog to make sure they actually work.
Our unit tests target #2 by exercising as much code as possible. We tend to apply the approaches of minimal API usage (passing as few arguments as possible), maximal usage (passing every argument possible) as well as explicitly targeting corner cases. We measure code coverage and for the key namespaces (libddog.dashboards, libddog.metrics) we aim for 100% coverage.
Passing unit tests are a good proxy for passing integration tests, because they are a superset of the same code under test.
Our type checks target #2 and both rely on, and validate, our persistent use of type annotations in libddog. Type annotations are also a key benefit for users of libddog, because their IDE can use them for code completion and highlight errors.
Our style checks target #3 to remain close to idiomatic use of Python and avoid common pitfalls in the language.
Our code formatter targets #3 by keeping code style consistent across the project in the library, tests and example code.
Our use of tox targets #4 and #5. When it comes to testing the ddog command line tool we use tox to exercise commands that don't require any dependencies (like listing definitions). This amounts to a smoke test.
When it comes to problems with our packaging we use tox to create a fresh environment, install libddog into it (exercising setup.py, MANIFEST.in and friends), and then run tests or ddog inside of that.
We also use tox to run ddog against a version of libddog that is installed directly from PyPI as part of our post-release checks. (The pypi-cli env in tox).
No testing method is perfect, but the more testing we do the greater the chance that we will catch problems.
| Method | How to run | Runs in CI? |
|---|---|---|
| Integration tests | ./integtests |
❌ |
| Unit tests | ./unittests |
✔️ |
| Unit tests under coverage | ./unittests_coverage |
✔️ |
| Static type checker | ./typecheck |
✔️ |
| Style checker | ./stylecheck |
✔️ |
| Code formatter | ./reformat |
✔️ |
| Tox | tox |
✔️ |
As much as possible we run our tests and checks in CI so that developers are alerted to problems as early as possible.
QA steps to run to make sure master is in a releasable state:
- Make sure the API is in a consistent state (there are no half implemented features).
- Make sure that all CI jobs are green.
- Run the integration tests and visually inspect the dashboards.
Create a commit for the release:
- Bump version in
libddog/__init__.pyto a final release version, eg.0.0.2. - Update CHANGELOG.
- Create a commit:
git commit -am'bump version to <version>' - Push the new commit:
git push
Perform the release:
- Git tag the new version:
git tag -a <version> -m <version> - Push the new tag:
git push --tags - Make sure there are no lingering distributions:
rm -rf dist/ - Create a source distribution:
python setup.py sdist - Create a binary distribution:
python setup.py bdist_wheel - Push the distributions to PyPI:
twine upload dist/* -u <username> -p <password>
Post-release checks:
- Run tox env to test the released version:
tox -e pypi-cli. Make sure the libddog version listed as installed is the version you just released. If not delete the tox envrm -rf .tox/pypi-cliand re-run.
If at this point you discover that the release is broken PyPI will let you yank/delete it and you can redo it. As long as this happens as part of the release process, it's fair to assume that noone is using the new version yet, so it should be okay.
By contrast, if you discover that an earlier released version is broken it's better to yank it than to leave it up for users to stumble over. Update the CHANGELOG as well to document this.
Finalizing steps:
- Bump version in
libddog/__init__.pyto an alpha release version, eg.0.0.3a0. - In the CHANGELOG create a heading for the next (upcoming) version so that changes can be added there incrementally.
- Create a commit:
git commit -am'bump version to <alpha-version>'