Blog

  • Monitoring imbalance prices with Home Assistant

    With the rise of home batteries trading on the imbalance market and solar panel features like Zonneplan’s PowerPlay, there’s growing interest in understanding how imbalance prices fluctuate throughout the day. Fortunately, websites like Tenergy and Tennet offer real-time insights into balance prices. For those looking to integrate this data into their smart homes, you can load these prices into Home Assistant and create graphs to visualize the historical and current imbalance prices.

    Setting up monitoring

    To start, fetch the real-time prices from Tennet. Use the script below, which runs a GET request every 60 seconds to load the latest prices from Tennet.

    - platform: rest
      name: Tennet Delta Prices
      resource: https://www.tennet.org/xml/balancedeltaprices/balans-delta.xml
      method: GET
      headers:
        User-Agent: Home Assistant
      value_template: >
        {% set ns = namespace(record='') %}
        {% for record in value_json.BALANCE_DELTA.RECORD %}
          {% if loop.first %}
            {% set ns.record = record %}
          {% endif %}
        {% endfor %}
        {{ ns.record | tojson }}
      scan_interval: 60
    

    Once you have the prices loaded, extract the relevant information. The prices can be positive or negative, depending on the grid balance. Below is an example script for extracting this data and storing it in a sensor.

    - platform: template
      sensors:
        tennet_balance_price:
          unit_of_measurement: "EUR"
          friendly_name: "Tennet Balance Price"
          value_template: >
            {% set record = states('sensor.tennet_delta_prices') | from_json %}
            {% if record.MIN_PRICE is defined %}
              {{ record.MIN_PRICE }}
            {% elif record.MAX_PRICE is defined %}
              {{ record.MAX_PRICE }}
            {% else %}
              N/A
            {% endif %}

    Restart your Home Assistant after setting up these sensors. You should now have a sensor called sensor.tennet_balance_price that can be used in a graph component (see below for visualization). Your output should look like this:

    Sometimes, the grid will have a major surplus or shortage of energy, indicated as “Emergency Power”. This usually results in a significant spike in imbalance prices. To monitor this, create an additional sensor:

    - platform: template
      sensors:
        tennet_emergency_power:
          friendly_name: "Tennet Emergency Power"
          value_template: >
            {% set record = states('sensor.tennet_delta_prices') | from_json %}
            {{ record.EMERGENCY_POWER if record.EMERGENCY_POWER is defined else 'N/A' }}

    Now, you’ll have an additional sensor called sensor.tennet_emergency_power that can be used in an automation to notify you when emergency power is activated.

    Visualizing imbalance prices

    Of course, you’ll want to use the data you’ve imported to visualize the prices and observe how they develop throughout the day. For graphs, I’m using the Apex Charts Card, a flexible library with many charting options. If you haven’t installed ApexCharts yet, follow the installation instructions first, or leverage the built-in history graph. Here’s an example of how you can visualize the imbalance prices using Apex Charts:

    type: custom:apexcharts-card
    apex_config:
      chart:
        height: 250px
    graph_span: 12h
    header:
      show: true
      title: Balance prices
      colorize_states: true
      show_states: true
    all_series_config:
      stroke_width: 2
    series:
      - entity: sensor.tennet_balance_price
  • Uploading your gas meter to MinderGas.nl with Home Assistant

    MinderGas.nl is a popular tool to get more insight into your gas usage. If you haven’t used MinderGas.nl yet, this tool allows you to get more insight into how much gas you’re using and lets you compare against peers and your historical gas usage. This is ideal if you’re trying to save gas and making adjustments in your house.

    When I started with this tool, I manually entered the values, but why would you do something by hand if you can automate it too. Since I’m using Home Assistant for my home automation (more on this soon!), it should be easy to automate this task using Home Assistant.

    I already have my gas meter available as a sensor in Home Assistant (sensor.gas_meter) thanks to my Toon thermostat that I integrated with Home Assistant. MinderGas.nl has an API that allows us to upload our gas meter values easily. 

    This guide has 3 simple steps we need to follow:

    1. Create an API token and store it safely in Home Assistant
    2. Add a rest command to your configuration that calls the MinderGas.nl API
    3. Add the daily automation that uploads the gas meter values

    Create an API token and store it safely in Home Assistant

    First of all, we need to create an API token to use the MinderGas.nl API. If you already have an account, you can go to https://www.mindergas.nl/member/api and generate an API token. Now, store this API token safely in your secrets.yaml file in Home Assistant. This file should contain:

    mindergas_api_token: <token>

    Add a rest command to your configuration that calls the mindergas.nl API

    This step creates the command that will call the API of MinderGas.nl. The command does not contain any information about the date or gas meter values yet. Those will be added later in the daily automation. 

    Add the code below to your configuration.yaml file:

    rest_command:
      mindergas_upload:
        url: 'https://www.mindergas.nl/api/gas_meter_readings'
        method: POST
        headers:
          content-type: application/json
          AUTH-TOKEN: !secret mindergas_api_token
        payload: '{ "date": "{{date}}", "reading": "{{reading}}" }'

    Add the daily automation that uploads the gas meter values

    To get the most detailed view of our gas usage, we want to upload our gas meter values every day. MinderGas.nl indicates that they see the values as the end values of that day, so we’ll want to upload the values as late as possible in the day. I’ve chosen to use 23:58 local time, giving the script a little buffer before the next day kicks in.

    To avoid making mistakes, I recommend creating this automation using the YAML editor or add it directly to the automations.yaml file:

    - alias: MinderGas Upload
      trigger:
      - platform: time
        at: '23:58'
      action:
        data_template:
          date: '{{ (as_timestamp(now())) | timestamp_custom("%Y-%m-%d", True)}}'
          reading: '{{ states(''sensor.gas_meter'') }}'
        service: rest_command.mindergas_upload
      mode: single

    Once enabled, the script will run each day at 23:58 and upload your gas meter values to MinderGas.nl. I want to thank jvdmast for his example script, as this gave me great guidance on how to get this working.

  • Environment Is Everything and Other Tips For Your Open Source Project

    In 2014, HackerOne launched their first open source project and two years in, we have 50 contributors, received almost 700 stars on GitHub, and are getting daily issues and pull requests. While collaborating with the open source community can be inherently time consuming, as users of open source software we see it as a great way to give back, collaborate with developers outside HackerOne and share what we are working on.

    One of the most important things to be successful is creating a friendly and open environment, being responsive on issues and pull requests, and making time to manage the workload. Open source projects don’t start as a community, but you can build one. In the early days we fixed many bugs ourselves, but nowadays we see that most bugs are fixed by others. That’s a big win for collaboration and exactly how we want this project to behave. Dedicating enough time in the beginning of the project will pay you back later on when you reach a critical mass. For example, React started as a company project by Facebook and Rails was once just a one man show by David Heinemeier Hansson.

    Create the best open source environment

    As the maintainer of an open-source project, you should provide your collaborators with a decent set of tools to work with. This ensures you’ll receive more pull requests with bug fixes and new features. Receiving external pull requests is the best thing that can happen to your project because it signals engagement.

    From the open-source projects I worked in, I have found the ones with the best development environments are the most attractive. In HackerOne’s Datepicker project, we started with a simple example page that loaded the external code into an HTML file that was hosted inside the project. Later on, we refactored this into a full React App which can be used as a starting point for your own local development work. Having a great development environment, together with a clean API are the key things to think of when starting your own project.

    Some things we did to help others to work on this project:

    • Linters — Everyone has their own preferences for coding, but by adding linters to enforce to most important things, you can make sure your codebase stays in a healthy state. In our case, we use JavaScript and CSS linters for syntax checks and enforcing a default code style.
    • Tests — You don’t want someone to submit a pull request with a bug fix that breaks something else. Aim for great test coverage (93% and counting at this moment) with the latest test frameworks. We use Karma as a test runner together with Istanbul to calculate the test coverage.
    • Continuous Integration — Use Travis or CircleCI to run automated tests, linters and coverage tools on your pull request. We’ve configured our coverage tool in a way that forces you to cover at least 75% of your new code with tests.

    Some things that we haven’t done yet, but are still on our list:

    • API — Be aware of feature bloat. Don’t have 4 features all doing nearly the same thing. Write a clean API which everyone can use and that extends to build new features. A simple API is easier to maintain and makes your project much more approachable to both consumers and contributors.
    • Documentation — Have the best documentation possible to make it easy for new users to discover the features and usage of your project. If you’re using React, use react-docgen to generate API documentation.

    What about bugs?

    Most open-source projects are side projects for the people who contribute to them. If it’s too time-consuming or too difficult, people won’t want to contribute. However, if you create a friendly and powerful environment for others to work in, the collaboration starts automatically.

    For HackerOne, one of the greatest benefits of using an open-source project is finding new bugs with the help of contributors. As others use our project people discover and fix bugs they encounter, contributing to a better component that we also benefit from.

    Be responsive–especially to your biggest contributors

    We can’t stress enough the importance of being responsive. You owe it to your project and contributors. You don’t want your project to die a slow death, which is exactly what it will eventually do if you don’t respond quickly to your most eager contributors.

    At HackerOne we noticed one of our contributors was especially involved. A few months ago, Rafee offered his help in the long term, so we made him a co-owner of the project. He now has direct push access and is helping make our project more successful.

    These types of collaborations are the benefit of open source. Thanks to a great development environment, we can focus on helping people collaborate and finding answers for people who have questions. I’ve seen many issues where people were helping each other without any assistance from us. This is the benefit of open source!

    Are you interested in participating in HackerOne’s Datepicker project? Go to https://github.com/Hacker0x01/react-datepicker and check out our issues page.

    This post was previously posted on the HackerOne blog.

  • How to secure your WordPress website

    Last year, HackerOne ran an experiment with a WordPress website. WordPress is a powerful platform with many options, including how you approach security.

    From our experiment, and the bug bounty program we ran in parallel, here are my top security tips for securing your WordPress site.

    Enforce a Content Security Policy

    A Content Security Policy (CSP), gives you the ability to block control all content sources on your website. Not nearly enough people are utilizing CSP with only 0.37% of Alexa top million websites use any CSP. Enforcing a CSP can help prevent cross-site scripting (XSS) and other related attacks. With this plugin, you can run the CSP in test mode first to see which requests are made while browsing your site. Please make sure you tune the config until you see no errors anymore. Now you can turn on your CSP.

    Google Apps Login

    The login page of your WordPress website is usually one of the weakest parts. By using Google Apps login, you can use the security standard of Google to secure your login page. You gain the ability to use two-factor authentication and to enforce strong passwords if you’re an Enterprise user.

    Use CloudFlare

    Cloudflare is a CDN/DNS provider that can boost the security of your website for free. Adding Cloudflare to your website gives you the ability to use some of their cool features like adding SSL to your website (for free!). Also, Cloudflare’s DNS support protects you from security threats and DDoS attacks.

    Offload your uploads to S3

    Most WordPress sites are hacked because of malicious data in their uploads folder. For example, hackers will upload infected PHP files to the uploads folder and will execute scripts that can use your website to spam others. By offloading uploads to Amazon S3, we make sure malicious data can never execute on our website.

    Be aware of content injections

    Most reports we received through our bug bounty program were about content injection. The severity of these issues depends on the content of your site and what can be injected, of course. There’s no real solution for these issues and we resolved them on a case-by-case basis. Check out our publicly disclosed content injection reports:

    Turn off XMLRPC

    XMLRPC is a WordPress feature that can be used for pingbacks and trackbacks. However, it can also be used for DDoS attacks. Since you don’t need this feature for day-to-day blogging, it’s better to turn this off. To turn XMLRPC off, add this to your theme’s functions.php:

    add_filter('xmlrpc_methods', 'remove_xmlrpc_pingback_ping');
    
    function remove_xmlrpc_pingback_ping($methods) {
     unset($methods['pingback.ping']);
     return $methods;
    }

    Keep your WordPress up-to-date

    Automattic, the company behind WordPress, runs a bounty program for WordPress. They are continuously resolving issues with security implications. It is important that you keep your WordPress installation up-to-date to ensure you are protected from known issues.

    These suggestions are based on the experience we had with running a bug bounty program for a WordPress website. If you have other tips, let me know in the comments section.

    Also, don’t forget to check out other cool guides like:

    Martijn Russchen, Product Manager HackerOne

  • You’re the product

    Laatst kwam ik de volgende quote tegen op Twitter:

    If you’re not paying for the product then you are the product.

    Een product waar je zelf het product bent? Dat klinkt natuurlijk als een slimme quote, maar er zit wel degelijk een kern van waarheid in. Kijk bijvoorbeeld naar Facebook of Twitter. Zij verkopen jouw data aan bedrijven, en op het moment dat jij het product niet gebruikt zullen zij niets aan jou verdienen.

    Internet gebruikers zijn gewend om veel dingen gratis te krijgen of te kunnen gebruiken. En dus zijn bedrijven ook genoodzaakt om manieren te vinden om geld te verdienen. Veel mensen zouden raar staan te kijken zodra Facebook ineens $10 per maand kost. Zelf zou ik het er graag voor over hebben, maar ik denk dat velen zullen afhaken op dat moment.

    Toch zijn er ook initiatieven waar mensen wel voor willen betalen. Het beste voorbeeld daarvan is toch wel De Correspondent. Maar hoe kan dat? Zijn mensen moe van de oppervlakkige berichtgeving op nieuwssites en zijn ze op zoek naar meer verdieping? Of is het gewoon toeval of heeft De Correspondent heel veel geluk?

  • Nieuwe startup? Vind een ambassadeur!

    Heb je een goed idee of ben je net begonnen met je startup? Misschien is het dan een tijd om een ambassadeur te gaan zoeken die jouw idee kan verkopen aan het publiek. Het klinkt misschien heel logisch, maar met een goede ambassadeur kom je een stuk verder.

    Een goed voorbeeld is Blendle. Mede dankzij de bekendheid van Marten Blankensteijn en Alexander Klöpping heeft Blendle een hoop bekendheid gekregen. Zo waren ze bijvoorbeeld te gast in de Wereld Draait Door en zijn er vele artikelen geschreven over het bedrijf. Dit terwijl er genoeg andere soortgelijke initiatieven zijn. Alleen zijn deze lang niet zo bekend als Blendle.

    Neem bijvoorbeeld het bedrijf Myjour. Dit bedrijf probeert hetzelfde te doen als wat Blendle doet alleen is dit bedrijf totaal niet bekend bij de mensen. Dat is goed te zien in de onderstaande afbeelding die een weergave geeft van het aantal zoekopdrachten naar beide bedrijven.

    Blauwe lijn: Blendle, Rode lijn: Myjour

    Is het succes van bedrijven als De Correspondent (Rob Wijnberg), Blendle en WeTransfer (Nalden) dan afhankelijk van hun ambassadeur? Dat is natuurlijk lastig te zeggen, maar duidelijk is dat een stuk makkelijker aan aandacht van de media komen. Om met je startup snel bekendheid te krijgen is het belangrijk dat je een ambassadeur gaat vinden die je product wil promoten en gebruiken. Een positieve ervaring door het gebruik zal een positief effect hebben op de promotie die de ambassadeur maakt.

    Promotie door een ambassadeur hoeft niet meteen te betekenen dat hij of zij op televisie je product moet gaan promoten, maar kan ook gedaan worden door bijvoorbeeld het gebruik van jouw product. Stel je voor dat een bekende Twitteraar alle linkjes naar artikelen via jouw platform laat lopen. Op deze manier komen heel veel mensen in aanraking met jouw product en bereik je ineens een grote groep mensen zonder actieve promotie. Op dat moment is het natuurlijk van belang dat jij ervoor zorgt dat je een viraal model ontwikkeld waarbij je die bezoekers weet vast te houden en zover krijgt dat ze zelf het product gaan gebruiken.

  • The annotated web

    Recently, MG Siegler of TechCrunch wrote about the 75-20-5 percentage rule: the observation that 75 percent of what you read in the tech press is somewhat accurate, 20 percent is complete bullshit, and 5 percent is actually true. The conclusion being that you cannot trust anything you read on the web without checking first. Maybe he’s right, but how would we check? Most of us don’t have the time, expertise or energy to check how accurate the information that we’re reading is. With the ever-increasing speed of new media, the need to post quickly seems to beat the need for accuracy. We’re moving towards a world where fact checking is disappearing.

    “I wish I had some solution. I don’t.”, writes Siegler. Luckily, several projects are aiming for a solution. The most well-known crowdsourced example is Wikipedia. Volunteers have built a shared frame of reference to capture the collective knowledge of everyone who participates. However, it’s a single website and does not give you information while you are visiting other websites. For this, we would need a tool that transcends websites, and pulls in information from different sources, as you are reading an article.

    Just imagine what an annotated web would look like. What if every website could be annotated by anyone, visible to everyone? That way, you could annotate and share why you agree or disagree with statements. Others could then read your notes. Sounds futuristic? Well, Marc Andreessen planned to have annotation be part of the browser when he was working on Mosaic. Unfortunately, it wasn’t technically feasible at that time.

    With modern technology, it has become feasible now to build such a system. Several initiatives like the Open Knowledge Foundation and Hypothes.is are working on the building blocks of an annotated web. At Factlink, our focus is on building a layer over the web that shows the credibility of the information. It’s time to start building a more transparent and accountable internet. We hope tools like Factlink can contribute to this. If you like this vision, please join the effort and check out these exciting projects!

    Recommended reading:

    This post is also published on our blog.