Posts

This post describes how to deploy cilium (and hubble) using docker on a Linux system with k3d or kind to test it as CNI and Service Mesh. I wrote some scripts to do a local installation and evaluate cilium to use it at work (in fact we are using cilium on an EKS cluster now), but I thought it would be a good idea to share my original scripts in this blog just in case they are useful to somebody, at least for playing a little with the technology. LinksAs there is no point on explaining here all the concepts related to cilium I’m providing some links for the reader interested on reading about it: What is CNI?What is Cilium?What is eBPF?What is Hubble?Why use Cilium with Kubernetes?...

This entry explains how I have configured a linux bridge, dnsmasq and iptables to be able to run and communicate different virtualization systems and containers on laptops running Debian GNU/Linux. I’ve used different variations of this setup for a long time with VirtualBox and KVM for the Virtual Machines and Linux-VServer, OpenVZ, LXC and lately Docker or Podman for the Containers. Required packagesI’m running Debian Sid with systemd and network-manager to configure the WiFi and Ethernet interfaces, but for the bridge I use bridge-utils with ifupdown (as I said this setup is old, I guess ifupdow2 and ifupdown-ng will work too). To start and stop the DNS and DHCP services and add NAT rules when the bridge is brought up or down I execute a script that uses: ip from iproute2 to get the network information,dnsmasq to provide the DNS and DHCP services (currently only the dnsmasq-base package is needed and it is recommended by network-manager, so it is probably installed),iptables to configure NAT (for now docker kind of forces me to keep using iptables, but at some point I’d like to move to nftables)....

This post describes how I’ve put together a simple static content server for kubernetes clusters using a Pod with a persistent volume and multiple containers: an sftp server to manage contents, a web server to publish them with optional access control and another one to run scripts which need access to the volume filesystem. The sftp server runs using MySecureShell, the web server is nginx and the script runner uses the webhook tool to publish endpoints to call them (the calls will come from other Pods that run backend servers or are executed from Jobs or CronJobs). Note: This service has been developed for Kyso and the version used in our current architecture includes an additional container to index documents for Elasticsearch, but as it is not relevant for the description of the service as a general solution I’ve decided to ignore it on this post. HistoryThe system was developed because we had a NodeJS API with endpoints to upload files and store them on S3 compatible services that were later accessed via HTTPS, but the requirements changed and we needed to be able to publish folders instead of individual files using their original names and apply access restrictions using our API. Thinking about our requirements the use of a regular filesystem to keep the files and folders was a good option, as uploading and serving files is simple. For the upload I decided to use the sftp protocol, mainly because I already had an sftp container image based on mysecureshell prepared; once we settled on that we added sftp support to the API server and configured it to upload the files to our server instead of using S3 buckets. To publish the files we added a nginx container configured to work as a reverse proxy that uses the ngx_http_auth_request_module to validate access to the files (the sub request is configurable, in our deployment we have configured it to call our API to check if the user can access a given URL). Finally we added a third container when we needed to execute some tasks directly on the filesystem (using kubectl exec with the existing containers did not seem a good idea, as that is not supported by CronJobs objects, for example). The solution we found avoiding the NIH Syndrome (i.e. write our own tool) was to use the webhook tool to provide the endpoints to call the scripts; for now we have three: one to get the disc usage of a PATH,one to hardlink all the files that are identical on the filesystem,one to copy files and folders from S3 buckets to our filesystem....

Since a long time ago I’ve been a gitlab-ce user, in fact I’ve set it up on three of the last four companies I’ve worked for (initially I installed it using the omnibus packages on a debian server but on the last two places I moved to the docker based installation, as it is easy to maintain and we don’t need a big installation as the teams using it are small). On the company I work for now (kyso) we are using it to host all our internal repositories and to do all the CI/CD work (the automatic deployments are triggered by web hooks in some cases, but the rest is all done using gitlab-ci). The majority of projects are using nodejs as programming language and we have automated the publication of npm packages on our gitlab instance npm registry and even the publication into the npmjs registry. To publish the packages we have added rules to the gitlab-ci configuration of the relevant repositories and we publish them when a tag is created. As the we are lazy by definition, I configured the system to use the tag as the package version; I tested if the contents of the package.json where in sync with the expected version and if it was not I updated it and did a force push of the tag with the updated file using the following code on the script that publishes the package: # Update package version & add it to the .build-args INITIAL_PACKAGE_VERSION="$(npm pkg get version|tr -d '"')" npm version --allow-same --no-commit-hooks --no-git-tag-version \ "$CI_COMMIT_TAG" UPDATED_PACKAGE_VERSION="$(npm pkg get version|tr -d '"')" echo "UPDATED_PACKAGE_VERSION=$UPDATED_PACKAGE_VERSION" >> .build-args # Update tag if the version was updated or abort if [ "$INITIAL_PACKAGE_VERSION" != "$UPDATED_PACKAGE_VERSION" ]; then if [ -n "$CI_GIT_USER" ] && [ -n "$CI_GIT_TOKEN" ]; then git commit -m "Updated version from tag $CI_COMMIT_TAG" package.json git tag -f "$CI_COMMIT_TAG" -m "Updated version from tag" git push -f -o ci.skip origin "$CI_COMMIT_TAG" else echo "!!! ERROR !!!" echo "The updated tag could not be uploaded." echo "Set CI_GIT_USER and CI_GIT_TOKEN or fix the 'package.json' file" echo "!!! ERROR !!!" exit 1 fi fi...

As promised, on this post I’m going to explain how I’ve configured this blog using hugo, asciidoctor and the papermod theme, how I publish it using nginx, how I’ve integrated the remark42 comment system and how I’ve automated its publication using forgejo and json2file-go. It is a long post, but I hope that at least parts of it can be interesting for some, feel free to ignore it if that is not your case …​ :wink: Hugo ConfigurationTheme settingsThe site is using the PaperMod theme and as I’m using asciidoctor to publish my content I’ve adjusted the settings to improve how things are shown with it. The current config.yml file is the one shown below (probably some of the settings are not required nor being used right now, but I’m including the current file, so this post will have always the latest version of it): config.yml baseURL: https://blogops.mixinet.net/ title: Mixinet BlogOps pagination: pagerSize: 5 theme: PaperMod destination: public/ enableInlineShortcodes: true enableRobotsTXT: true buildDrafts: false buildFuture: false buildExpired: false enableEmoji: true pygmentsUseClasses: true minify: disableXML: true minifyOutput: true languages: en: languageName: "English" description: "Mixinet BlogOps - https://blogops.mixinet.net/" author: "Sergio Talens-Oliag" weight: 1 title: Mixinet BlogOps params: homeInfoParams: Title: "Sergio Talens-Oliag Technical Blog" Content: >  taxonomies: category: categories tag: tags series: series menu: main: - name: Archive url: archives weight: 5 - name: Categories url: categories/ weight: 10 - name: Tags url: tags/ weight: 10 - name: Search url: search/ weight: 15 outputs: home: - HTML - RSS - JSON params: env: production defaultTheme: light disableThemeToggle: false ShowShareButtons: true ShowReadingTime: true disableSpecial1stPost: true disableHLJS: true displayFullLangName: true ShowPostNavLinks: true ShowBreadCrumbs: true ShowCodeCopyButtons: true ShowRssButtonInSectionTermList: true ShowFullTextinRSS: true ShowToc: true TocOpen: false comments: true remark42SiteID: "blogops" remark42Url: "/remark42" profileMode: enabled: false title: Sergio Talens-Oliag Technical Blog imageUrl: "/images/mixinet-blogops.png" imageTitle: Mixinet BlogOps buttons: - name: Archives url: archives - name: Categories url: categories - name: Tags url: tags social: true socialIcons: - name: CV url: "https://www.uv.es/~sto/cv/" - name: Debian url: "https://people.debian.org/~sto/" - name: GitHub url: "https://github.com/sto/" - name: GitLab url: "https://gitlab.com/stalens/" - name: Linkedin url: "https://www.linkedin.com/in/sergio-talens-oliag/" - name: RSS url: "index.xml" assets: disableHLJS: true favicon: "/favicon.ico" favicon16x16: "/favicon-16x16.png" favicon32x32: "/favicon-32x32.png" apple_touch_icon: "/apple-touch-icon.png" safari_pinned_tab: "/safari-pinned-tab.svg" fuseOpts: isCaseSensitive: false shouldSort: true location: 0 distance: 1000 threshold: 0.4 minMatchCharLength: 0 keys: ["title", "permalink", "summary", "content"] markup: asciidocExt: attributes: {} backend: html5s extensions: ['asciidoctor-html5s','asciidoctor-diagram'] failureLevel: fatal noHeaderOrFooter: true preserveTOC: false safeMode: unsafe sectionNumbers: false trace: false verbose: false workingFolderCurrent: true privacy: vimeo: disabled: false simple: true twitter: disabled: false enableDNT: true simple: true instagram: disabled: false simple: true youtube: disabled: false privacyEnhanced: true services: instagram: disableInlineCSS: true twitter: disableInlineCSS: true security: exec: allow: - '^asciidoctor$' - '^dart-sass-embedded$' - '^go$' - '^npx$' - '^postcss$' Some notes about the settings: disableHLJS and assets.disableHLJS are set to true; we plan to use rouge on adoc and the inclusion of the hljs assets adds styles that collide with the ones used by rouge.ShowToc is set to true and the TocOpen setting is set to false to make the ToC appear collapsed initially. My plan was to use the asciidoctor ToC, but after trying I believe that the theme one looks nice and I don’t need to adjust styles, although it has some issues with the html5s processor (the admonition titles use <h6> and they are shown on the ToC, which is weird), to fix it I’ve copied the layouts/partial/toc.html to my site repository and replaced the range of headings to end at 5 instead of 6 (in fact 5 still seems a lot, but as I don’t think I’ll use that heading level on the posts it doesn’t really matter).params.profileMode values are adjusted, but for now I’ve left it disabled setting params.profileMode.enabled to false and I’ve set the homeInfoParams to show more or less the same content with the latest posts under it (I’ve added some styles to my custom.css style sheet to center the text and image of the first post to match the look and feel of the profile).On the asciidocExt section I’ve adjusted the backend to use html5s, I’ve added the asciidoctor-html5s and asciidoctor-diagram extensions to asciidoctor and adjusted the workingFolderCurrent to true to make asciidoctor-diagram work right (haven’t tested it yet)....