Development Updates info.btree.at

Hi all,

after my unwillingness to write more tests to prevent burn-out. I did add to the server side the public API for hive scales and iCal.

As for hive scales, it was a lot easier to manage basic validation thanks to express-validator and bruteforce, timelimit and db post was rather quickly set up in comparison to my old PHP self-made behemoth.

Example of my route validation for hive scale api calls:

Validator.validate([
          param('ident').isString(),
          param('api').isString(),
          query('action').exists().isString().isIn(['CREATE', 'CREATE_DEMO']),
          query('datetime').optional().isISO8601().toDate(),
          query('weight').optional().isNumeric().toFloat(),
          query('temp1').optional().isNumeric().toFloat(),
          query('temp2').optional().isNumeric().toFloat(),
          query('hum').optional().isNumeric().toFloat(),
          query('rain').optional().isNumeric().toFloat(),
          query('note').optional().isString().isLength({ max: 300 }),
        ]),

Generating iCal endpoints was exceptional easier than I used to do it, thanks to ical-generator. There are really plentiful of node packages and Iā€™m really happy for my decision to write the server side on node.

Example of my the user API page on the frontend, which Iā€™m proud of myself how clean I managed to get it:

On the frontend part I started with creating statistics which I really enjoy. First I wanted to create everything with d3 but now using echarts as it is better integrated into vue.

Here a small video of the current implementation:

Lastly currently Iā€™m in alpha stage and some users already testing the app. If anyone feels like giving feedback and doing a little bit of ā€œbugā€ hunting send me a message (this is not meant as public beta, so if you only want to check out the app please be patient as I will do a public beta in the near future).

Cheers
Hannes

1 Like

Hi all,

little bit lacking with my development speed, due to more work in the forest. In addition we had a windstorm a few days ago, which means even more workā€¦

Nevertheless, after some crucial bug fixes I now opened the new version for open beta testing. If you are already an user of my application you can login with your credentials. The database is a copy of today (2022-08-22) and it is not connected to the live app.

Login: https://beta.btree.at/

In addition I created a test user, as you cannot register a new account on the beta page (my email logic is not finished).

Test-User: messe@btree.at
PW: messe_btree

Theoretically you could delete the demo account, I did not care to protect it. If the demo account is broken, please donā€™t hesitate to tell me. Then I need to reset the database.

Cheers
Hannes

Hi all,

the past few days I did a redesign of my info page, which will again hold documentation for my application and some personal related stuff.

My previous version was made with Nuxt (framework for Vue), but the overhead always felt a little bit high and I did not like the SPA feeling for a simple static page. On the search for alternatives I stumbled on Hugo, VitePress and Astro.

  • Hugo: After some messing around, it was a little bit too much effort and learning too much new stuff for me.
  • VitePress: Would be the quickest transition but did not want again an SPA

Finally I looked into Astro, the ā€œnewā€ kid on the block and recently reached a stable version 1.x.x. They promise speed by using an island design strategy and you can implement more or less all JavaScript frameworks you like. It also supports content in Markdown format without any additional plugins.

After some initial starting problems how to use components inside markdown files (hint: you need to use .mdx files). It was a very smooth transition from my Nuxt page to Astro.

The Markdown files were quickly set up with the correct frontmatter. An overall documentation template style was quickly adapted to my needs and the integration of my Vue components did work mostly without too much hassle.

After a few days of coding everything was running with auto deployment on my server via GitHub actions. The page runs perfectly fine and with incredible speed. I still have hydration errors, because of my dark mode and language Vue component (which I have no idea how to fix it), but donā€™t really mind the flickering so far.

Conclusion: Page speed is impressive, easy to adapt framework, nice documentation.

GitHub Repo: GitHub - HannesOberreiter/btree_info: Landing Page for Beekeeping App and Personal Homepage, made with Astro
Homepage: https://info.btree.at/

Cheers
Hannes

Many thanks that you share this journey with us! And respect that you change the horse while running, often people stay on the tools they are used to use because they use them :-) so sometimes not a good reason.

Hi clemens,

it is probably a selfish reason that I write here, as it gives me time to think what I archived so far and when writing it I often get more or better ideas. Explaining the problem is the first step to problem solving :nerd_face:.

In web dev you probably cannot adapt fast enough to new things anyway, but with astro I feel really good as it connects all the frameworks and you can easily switch out parts. Something similar I have written at the beginning when deciding on my frontend (9th March, 2022):

Selecting a framework for frontend could be easy but if you research the various javascript frameworks you kinda feel overwhelmed, as you donā€™t want to invest time into a technology which may be obsolete again in a year. The biggest out there are probably React and Angular and I played around with both, but it felt too big for me when starting with it. After the big ones I looked at ā€œfresherā€ starts Svelte and Vue.js, both felt very good with a lower entry barrier in my opinion.

The second reason why I like to write here about my journey is that it gives me a history of whats changed. Fun fact by pure chance I recently found some images of my very first version, which I presented in 2015. It was made with jQuery, PHP and self written CSS. I did even write a secondary version with jquery-mobile as framework. Crazy first steps into programming, when I think about it.

Sometimes I question my own sanity, that I keep on programming on my app in my free time. The reasons are a few users, which started using the app since the start and are still using it. To make something people use fills me with joy, I donā€™t know why but it is as it is.

Cheers
Hannes

3 Likes

Hi all,

working on my final touches before going live with the newest version, which should be this month. As I really want the new version running before the 2. eurobee in Germany, where I will be present.

CORS & Open API

Hate it or love it CORS is a pain. For security reason I only whitelist certain addresses to access my backend API. This caused some headaches with my open API for iCal and hive scales. One would thinks it is easy to lift CORS logic for only one route but it was harder than I though. Nevertheless after some trail and errors I finally managed to get it running in a somewhat clean way.

I use the node cors package and inject it into my main app at the beginning as global middleware, as I have many routes and use containers I did not want do explicit set CORS for each individual route. Therefore I needed a way around that. I somewhat did not read that you can set the CORS settings asynchronously but after I found out that you are able to do it, I came up with this solution:

// Inject cors middleware globablly and add settings as function
app.use(Cors(this.corsOptionsDelegate));
  /**
   * @description Configuring CORS asynchronously, will disable CORS for /external/ route
   */
  private corsOptionsDelegate = function (req, callback) {
    const origin = req.header('Origin'); // before this I have a custom middleware which always sets an origin, also if not given
    // Main settings
    let corsOptions = {
      origin: true,
      methods: ['GET', 'POST', 'PUT', 'PATCH', 'DELETE'],
      allowedHeaders: [
        'Accept',
        'Content-Type',
        'Authorization',
        'Origin',
        'From',
      ],
    } as any;
    let error = null;

    if (req.url.indexOf('external') !== -1) {
      // Allow API calls to scale and iCal without CORS
      corsOptions = { origin: false };
    } else if (
      authorized.indexOf(origin) === -1 &&
      origin &&
      env !== ENVIRONMENT.development
    ) {
      error = notAcceptable(`Domain not allowed by CORS: ${origin}`);
    }
    callback(error, corsOptions);
  };

Now everything and everyone should be able to push data or receive data from my API. I also updated the documentation, to reflect the changes for the new beta version. As for hive scale data, I simple use a GET request with the transferred data as params. I think this is the easiest solution and can be quickly adapted by users, as they simply need to change the url and add variable params in their code.

In addition I added a simple weight graph for the hive scale data, nothing too fancy and not compare able to the hiveeyes graphana dashboards, but good enough for me.

QR Codes

I did have the ability to scan and write QR codes in an older version of my app but discontinued it. Recently I checked again the usability and it is a lot better now. Therefore I create the option to scan and create QR codes for your colonies. The scanner works on all devices I tested (laptop mac, iPad, Android) and is very fast. Although, I simply save the url to the detail view of colonies, it has an advantage function to automatically select scanned hives if there is a hive dropdown on the current window. This should allow very fast working in the field if you have a couple hundred hives and donā€™t want to scroll through the list. It now feels even faster than NFC which was previously my goto option. But the big difference is still that the NFC scanner is always active in the background (if you activate it) and I can simply scan the tag. For the QR tag I need to open the camera dialog. There are pros and cons as with everything in life.

Cheers
Hannes

1 Like

Hi all,

last week I attended the aquaculture conference in Italy, Rimini and saw a lot of parallels to beekeeping, but they make more effort in topics like sustainable, environmental friendly and fish wellbeing, which I really liked. As vegetarian I think we should not farm fish anyway, but yeah aquaculture does also include stuff like algae farming. Of course the most interesting for me where the applied computer science and robotics talks.

Nevertheless, yesterday was a long day for me, the live migration to my new application version was happening. As this is my third ā€œbigā€ migration I already knew how to prepare myself.

This were my steps, which helped me a lot:

  • notify users inside the old app about the time of migration (~1 month before)
  • create a list in order you need to do for migration, eg. set old app into maintenance mode, reload live server with all new packages, create DNS and redirects ā€¦
  • health tips: yoga in the morning and prepare lunch the day before migration
  • backup backup backup and be ready to reroll in case of failure

Overall everything worked as expected, the database after migration needed again a ā€œrepairā€/ā€œoptimisationā€ but I was already prepared for it:

mysqlcheck -u -p -h -P --auto-repair --optimize --all-databases --verbose

Had some docker container starting failures due to missing env entry. Which did me cost some nerve to find but was ok after a few minutes of sweating.

After the service went online again and DNS was already changed, a user reported he could not log into the app. This was quite frustrating as I could not figure out why, but lastly the user itself gave the hint.

In the new version I clean the emails with normalizeEmail from express-validator. I did not know but gmail addresses with points eg. max.muster@gmail.com will be normalised to maxmuster@gmail.com. Of course after normalisation the new email is not the same as the registered email because in my old app I did not normalise it. Therefore, the user could not log into the app and could not reset his password. Therefore a quick decision was made to not normalise emails at all (the same as in the old app), this may cause pain in the future if one user mixes up upper case or lower case.

Next pain were the AppStore and PlayStore apps, I would love to drop them completely but lots of people want it and simply donā€™t know how to install PWA. Android was quickly archived and everything works more or less. iOS was a lot more pain, also because I really have zero experience with XCode and swift programming language. In addition iOS does not really like the kind of wrapper apps and donā€™t support PWAs. One main thing what did break the iOS app was printing, I solved it by disabling it. Not the best approach but yeah. Maybe someone here is more into swift programming? Currently the code looks like this:

 // redirect new tabs to main webview
 func webView(_ webView: WKWebView, createWebViewWith configuration: WKWebViewConfiguration, for navigationAction: WKNavigationAction, windowFeatures: WKWindowFeatures) -> WKWebView? {

        if (navigationAction.targetFrame == nil) {
            let app = UIApplication.shared;
            if(navigationAction.request.url?.absoluteString != ""){
                let url = navigationAction.request.url!;
                if !app.canOpenURL(url){
                    webView.load(navigationAction.request)
                }
            } else {
                // Notify user that printing is not allowed (blank new window, without url)
                self.notif()
            }
        }
        return nil
    }

Another problem caused the Stripe payment which I know solved by redirecting to the external browser, which is not the best but was my only viable solution:

if(requestUrl.absoluteString.contains("stripe")){
  UIApplication.shared.open(requestUrl)
  decisionHandler(.cancel)
} else {
  decisionHandler(.allow)
}

Today after the big migration was more relaxing finishing new demo images for Play and App Store and cleaning some junk codes.

Cheers
Hannes

2 Likes

Hi all,

the 2. eurobee fair is over. Had a fun time but it reminded me that Iā€™m no sales person and donā€™t like places with a lot of people :smiley:.

For my master thesis presentation a few people showed up. I started with only 3 but the amount of listeners gradually increased, which made me quite happy.

Cheers
Hannes

2 Likes

Hi all,

just wanted to give some project updates and insights into my decision/problems etc.

Session storage changed to Redis

Previously I used my MariaDB as session database, which results in a lot of read and writes. Not that it really matters for my few active users, but I felt like a perfect place to over engineer. Therefore I changed to a Redis database for session storage only, it was my first time working with Redis thx to libraries like github.com/luin/ioredis it was actually fairly easy.

As I did not wanted to drop any session when changing storage I had a script to write the mariaDb session to Redis, this was really overkill as probably no one would mind to be dropped out of the app once but was an experience by itself and gave me a better understand for it. As GitHub (never) forgets here is the old commit were I deleted the code to transfer the session: commit/f101de207eacf3b6dbf6a3926aefa59b5e0f1029

Removed Browser Outdated Information

I though it is a nice feature to remind users that their browser is outdated, therefore I had a simple third party script installed to inform them with a fancy popup browser-update.org/. But it seemed that my users were more irritated by it and sadly by its own story Android phones which gets no upgrades would always see the update notice. Which is a common problemā€¦ therefore I removed this feature.

PWA

I have to say I still love PWA (progressive web apps) but it only works great with chromium browser and to be honest its a mess by itself. In the last update for iOS they made a few changes for PWA support, but it broke my app for a few users completely. Handling outdated content was not working anymore and users were stuck in an endless loop. In addition workbox which I use for building my PWA is in a little bit stale mode, it is made by google but did not get any love recently. This may change as discussed in a GitHub issue but my hopes are slim: GoogleChrome/workbox/issues/3149.

Nevertheless, my current workaround is to have PWA disabled for all iOS devices. As I only use a few PWA features it does not hurt. PWA was my road down for supporting full offline mode, but I probably will never archive this anyway.

LLM, OpenAI

Last but not least. Today I had a full free coding day and wanted to play around with an idea which I had a long time ago. Adding my documentation in front of an LLM to answer user questions. The idea itself is nothing new and a few companies already have it implemented on their homepage. Still wanted to ride the band wagon a little bit.

Most guides are in python, therefore I set up a small playground for myself: github.com/HannesOberreiter/wizbee

Probably the only ā€œfancyā€ difference is that I use Redis as my vector database, as I had such a good experience with it. After getting my documentation, ready and pasted into the vector database. I also connected some public available online resources on beekeeping. With this help the bot can also answer beekeeping specific answers.

The playground was easy to setup, thankfully as there are already lots of tutorials and using github.com/hwchase17/langchain. Getting it to work on my live node server and good in different languages took me on the other hand quite a bit, as I could not find any Node&Redis&Vector guides and the Redis documentation was a little bit over my head sometimes. But somehow I finished it today, but also a little bit burned out. Here is my class for prompt engineering and fetching from the vector database: btree_server/blob/main/src/api/services/wizbee.service.ts

Here is a short demo video:

Cheers
Hannes

2 Likes