Retrospecting on my first year at endjin
I am now over one year into my software engineering apprenticeship at endjin, and now an Apprentice Engineer II. As in my 1-month and 6-month retrospective blog posts, I will be sharing my experiences from the past year and using this post as an opportunity to introspect and look to the future.
The first 6 months – a recap
As I spoke about in my first two retrospective posts, the first 6 months of my apprenticeship was largely characterised by a sense of training – preparation before beginning to contribute. The first year of an apprenticeship at endjin is designed to have a strong training component, with half of an apprentice's time dedicated to training; some of the topics I spent time learning during the first 6 months included C# fundamentals, GitHub fundamentals, Azure fundamentals, ASP.NET, the Web, and others.
In addition to dedicated training time, I spent some time shadowing and pairing with other endjineers on both client and internal projects. There was also a software development project around updating an internal piece of software that me and the other apprentices worked on.
Towards the end of that first 6 month period, there was a push to get the apprentices more involved in ongoing projects, including client facing ones. The first client project that I was actively involved in (i.e. contributing to) was around load testing; the client wanted to understand the performance of their web application under load, and improve that performance to be able to onboard more customers and have the app work smoothly with increased traffic. My role within this effort was focused on the UI tests used to simulate a human interacting with the website. Through this I got to learn WebDriverIO, which is a JavaScript web testing framework, and as a result got an introduction to writing JavaScript code.
I also spoke about blogging, presenting and figuring out my interests across the different aspects of endjin's technical work (DevOps, engineering, data, etc) in my 6-month retrospective post. I will share the developments from the last 6 months in relation to those topics, as well as others, in this post.
The last 6 months
Reading Programming C# 8.0
One thing I spoke about in my 6 month retrospective was returning to dedicated C# training by reading Ian's Programming C# 8.0 Book. I completed an introductory C# course on Pluralsight in the first few months of my apprenticeship, and once I had finished reading The Pragmatic Programmer I needed a new book to continue my reading habit with.
I made some progress with reading Ian's book, however my daily reading habit fizzled out which basically killed my progress. I was going about reading this book the wrong way – you can't just read a technical book like this to learn the content, you have to implement the code samples yourself and experiment with them, etc. Another thing that contributed to the downfall of the habit was my explicit decision somewhere along the way to concentrate more on blogging. Now, there's no reason why these two things can't be done in the same day – there's plenty of working hours in the day, but I found, at the time, that the best time in the day for me to do reading and blogging is in the morning, preferably before 9am; as a result I allowed blogging to displace my reading habit.
It's a shame I did this because, as I mentioned in my previous retrospective post, I still didn't feel particularly confident with C# at the end of the first 6-months, and I wanted to rectify that. I largely feel the same way now, and now I'm even more convinced about the importance of having a strong grasp of a programming language, not only for the purpose of being able to program competently in that language, but for the general software engineering training you get as a result. More on this later.
Blogging
Over the last 6 months us apprentices have been strongly encouraged to do more blogging – setting a target of one blog per week. I mentioned in my 6-month retrospective post my thoughts on how writing can be used as a tool for learning since it forces you to think and question assumptions, I also expressed my desire to do more of it. Despite that, I'm quite disappointed with my blogging output over the last 6-months. An obvious set of reasons for the low output is that blogging is hard (I think) and time consuming, and I'm not particularly well practiced in writing articles. Another part of the explanation is that I tend to scope articles too widely and allow the scope to increase as I write, this causes articles in progress to drag on, which can affect motivation to finish. So, that's something I need to work on.
All that being said though, I think the time taken for me to write new blogs is trending downwards, which is good; I think it's simply a case of practice: the more I practice writing, the more efficient and productive I'll be. I'm also really pleased with some of the blogs I've written, for example the Computer Networking Essentials for Developers series and Understanding the Stack and Heap in C# (which got 14k views!).
Training
The percentage of my time spent on dedicated training over the last 6 months has decreased relative to the first 6 months of my apprenticeship, this is fine and expected given I've become more involved in project work and naturally spending more time on that.
I think as time goes on my training schedule will be increasingly dictated by active project work and the demands of the business. However, I think regular dedicated training time is still important, and an apprentice engineer II is still expected to spend a considerable chunk of their time on training. I also enjoy training, and learning new things in general; it's now about being able to balance dedicated training time with increased project involvement and responsibility.
The topics I've covered in my training time over the last 6 months include: PowerShell basics, debugging in Visual Studio, GitHub Actions, some Python bits-and-bobs, Bicep, and I recently finished reading The Phoenix Project.
PowerShell is probably part of the group of core technologies that we use at endjin. I wasn't particularly satisfied with the course I took and I plan to spend more time exploring PowerShell further. The Visual Studio debugging course was quite nice, it introduced me to many debugging features that I didn't even know existed.
GitHub Actions is GitHub's CI/CD platform, I've been spending some time with this over the last few months as part of an internal bit if work around improving the local dev experience when programming with PySpark, and Python more generally. endjin do quite a lot of work in Azure Synapse for data-oriented work, but developing programs in Synapse notebooks is quite clunky; using VS Code's tasks feature and some PowerShell scripts, we developed the ability to upload local Python packages and run local Python files on Synapse with a couple of clicks inside VS Code. Anyway, as part of this piece of work, we used GitHub Actions to setup a CI/CD pipeline for endjineer's to use when developing PySpark and regular Python packages. Naturally, as part of all this, I was doing some Python reading and experimenting too. For example, understanding OOP in Python, which is quite different to C#; getting familiar with type hints; and getting my head around packages and modules in Python, and the common patterns, best practices and tools used around this (namely Poetry). I produced a blog out the back of all this on CI with GitHub Actions and Python, and there's another in progress on CD with GitHub Actions and Python.
In addition to GitHub actions, Bicep and The Phoenix Project form a bit of a DevOps streak in my training from the last 6-months, which is no surprise since an interesting DevOps project kicked off at endjin in this time period which piqued my interest. Bicep is an infrastructure as code (IaC) domain specific language (DSL) that allows you to programmatically provision and manage resources in Azure. IaC is pretty powerful and is a key part of DevOps in the cloud. I really enjoyed covering the Bicep Fundamentals learning path on MS Learn, and I'd like to pick-up the intermediate and advanced versions at some point. The Phoenix Project is a book that teaches and demonstrates the power of DevOps principles through a fictional story. It's a great book that's easy to follow; it didn't concern itself with too much with particular technologies, which is good, rather it focuses on the core principles of DevOps, which are fundamentally around how to deliver value to customers through software.
A client project
After my involvement in the load testing project ended, I got involved in a newly commencing client project around automating a manual process involving Excel, a third party web service, and the client's internal systems. The process was time consuming and, given the fact it was being ran every day, expensive to the client. So, the goal was to automate the process end-to-end to reduce costs and free-up the time of the person executing the procedure.
The first part of the project was around discovery (although I wasn't involved much in this stage) – understanding the manual process and the needs of the client; afterwards was estimation and technology choice, some considerations around technology choice were that it should be able to easily integrate with the client's current MS Office based systems and match their technical maturity. Given that, some of the technologies being considered in the Microsoft/Azure space were Power Automate and Azure Logic Apps. We ended up going for a combination of Power Automate, for it's closeness to MS Office, and Azure Data Factory for some of the data transformation elements.
As the project progressed it was becoming increasingly apparent that this particular problem was outside the scope of the chosen technologies. For example, for automating the process of inserting data into excel spreadsheets, we frequently encountered time-out and data size issues due to Power Automate's (understandable, given it's a low/no code tool) limits; also, we found ADF to be too cumbersome for the task of communicating with the web service and processing the XML documents that were being returned by it. Consequently, we were forced into introducing different technologies and methods as time went on, increasing the complexity of the solution. For example, for the purpose of communicating with the web service, we used a Python-based Azure Durable Functions architecture, we also developed Python programs, that ran in those Functions, to parse and wrangle the XML data into the required format. We also used some Python code to resolve the issues and general slowness experienced with using Power Automate to manipulate Excel files.
Despite all of that, I quite like Power Automate and I can definitely see how it can be used to help automate many common business operations and provide good value, I think the particular problem we applied it to was just too far outside of its intended scope. I also wrote a blog post on the topic – demonstrating how Power Automate, along with Office Scripts, can be used to automate a basic manual reporting process in Excel.
Something else to note regarding this project was my definite increased responsibility relative to that on other earlier projects, demonstrated by, for example, me being responsible for some development tasks (developing Power Automate flows, fixing issues in Power Automate flows, and some Python bits) and, particularly, for being responsible for producing weekly updates for the client. The weekly updates, or ‘end of week updates' as well call them, consist of a slide deck and a video recording – you create a slide deck communicating the week's progress, issues, etc, and then produce a screen recording of you talking through the slide deck. I initially hated producing these. For the first couple I would constantly fumble and have to re-record the slide – very frustrating, however I kept going with them, and after probably three or four I was pretty comfortable. By the end of the project, I had done 10+ end of week updates, I had my own process which I was comfortable with, and my time taken to record went down from around 2 hours, for the first one, to around 15 minutes towards the end of the project. I'm quite pleased with that progress, it shows the power of a little perseverance and practice.
Azure Weekly
A recurring theme over these past 6-months has been my involvement in producing endjin's weekly newsletter on Azure: Azure Weekly. Endjin have an in-house content platform used to curate and compose our two weekly newsletters: Azure Weekly and Power BI Weekly; and, basically, every week for the last seven months or so, I've spent around 3 hours helping to select articles that should feature in the newsletter.
This has involved scanning and reading a lot of blog posts, and doing so provides a number of benefits. First, it provides a view into the many different facets of Azure and the technology space more generally, as a result I've come across activities, technologies, people and opinions that I otherwise would likely never have come across (or at least not this early). This helps you to see the bigger picture. It has also helped me to become more familiar with the dizzying number of services in Azure. At the beginning I was having to constantly search for terms online - I still have to a little now, such is the vastness of Azure. Another benefit of reading lots of articles is that it helps you with your own blogging.
Team meetups
Another recurring theme over the last 6 months has been the company meetups. When in regular working order, the endjin team meetup about every seven weeks. However, given the intermittent COVID situation during the first 6 months of my apprenticeship, none of the usual ones went ahead. There was one big meetup in Cambridge around when me and the other apprentices first joined, in which we did no work, had a huge meal and went out for drinks afterwards, but that was a one-off. Over the last 6 months the company has returned to its regular meetup cadence, all of which have been in London.
During the meetups we try to stick to activities that are different to the day-to-day, and also try to capture some aspects of regular office-based work that are harder to do, or are fundamentally not possible, as a remote company. I'm thinking here of live in-person presenting, or group discussions around strategy, the direction of the company, and the like – the types of discussions were you need free-flowing conversation, were people can interrupt one another easily, and so on (i.e. things that are less easy to do in Teams calls). The day's agenda is also pretty flexible to keep things more free-flowing and spontaneous, this is one aspect of in-office work that I think you miss out on with remote based work; in remote based work you mostly speak to people during organised meetings which have a topic and a purpose.
In addition to work related activities, we go out for a large lunch, and at the end of the work day, we go out to one of London's drinking/activity establishments. For example, at the last meetup we did crazy golf followed by more food and drinks, at the meetup before last we did a kind-of air hockey game with drinks and pizzas.
I really enjoy the team meetups and think they're important, not only to experience some aspects of in-person work that you miss out in remote work, like I mentioned above. There's the increased social nature with colleagues, which is important; I also think they allow the company to pause and take stock of where we are, and where we're going, which is super useful.
Recent internal projects
Most recently, together with the other apprentices, I have been focused on a number of internal IP-related projects. The main one being around the broad goal of improving the documentation for the various endjin-owned open source projects and associated libraries. Within that goal, we have been focusing on Corvus, which is a collection of fairly low level C# libraries used by other endjin-owned projects like marain and menes. This involves understanding the libraries, generating useful code examples, method descriptions, walkthrough videos and blog posts. Which is great because it means we get to develop our C# skills and contribute at the same time.
The other pieces of this recent IP-related work have been around developing endjin's internal collection of reusable Bicep modules, and investigating some of Azure's more recent offerings for their use in data-oriented projects. On the latter of those two, I've spent some time with Barry over the last month-or-so looking into Azure Synapse database templates and, more recently, Azure Purview.
Looking ahead
Blogging
As I spoke about earlier, one big area of focus for me is on increasing my blogging output. To produce blogs more regularly I have to write regularly, to write regularly I have to build habits that make regular writing a main feature of work. The idea of building habits is closely related to the topics of time management, prioritisation and productivity, which I've also been thinking about more lately.
Time management, prioritisation and productivity
I mentioned that I have felt a definite increase in responsibility over the last 6 months, this is expected and I think it will continue as I progress through the apprenticeship programme. A natural consequence of this is that you begin to think more about time management, how you prioritise tasks, and your productivity, and take these things more seriously. These things have been brought to the fore lately and emphasized to me and the other apprentices, for example James has suggested that we openly set out objectives at the beginning of the week and commit to them. Doing things this way definitely helps me. It's easy to have vague notions about what you want to do at the beginning of the week, but they're unlikely to be completed this way. I think time management and prioritisation are skills, just like any other, and I intend to work to improve them going forward into my second year at endjin.
Another aspect within the theme of ‘getting things done', is understanding your own brain and how it works best. This is definitely an important piece of the puzzle and another thing I've been thinking about a bit lately. For example, when is the best time to do tasks that require intense focus, like learning something new? (early morning time, for me). Or, what's the best way to spend the hour after lunch, when there is a slight slump mental sharpness?
Developments in my thoughts on career direction
I spoke in my 6-month retrospective post about me not yet having any particularly strong opinions on a preferred technical aspect of the work we do. That's still the case, pretty much. I joined endjin with a strong interest in data science, and whilst that's still true, in the whole, my training and activities over the last 6 months have not been much related to data science. And that's fine, I've realised that I quite like DevOps – hence taking the Bicep course and reading The Phoenix Project – and I've taken more of an interest than I expected in the more pure software engineering side of things.
I'm pretty relaxed about this. I plan to keep following my interests and see where they take me. Although, with that being said, I am conscious that it's important to specialise in some areas rather than know a bit about everything.
Getting to grips with C#
Like I mentioned earlier, one thing that has come to the forefront of my mind recently is the importance of having a strong grasp of a programming language. I think doing so helps to give you a broad software engineering education; your ability to solve problems with programs improves, you learn how to solve classes of problems that show up repeatedly. In addition, it's clear that many principles and patterns are common across programming languages and technologies, meaning that after learning one well, it's easier to pick up others. As such, a major goal of mine as an apprentice engineer II is to really get to grips with C#
Overall, I have thoroughly enjoyed my 1st year at endjin. I'm look forward to continue doing so and contributing more to the team in my 2nd year.
