When I was in college, I was required to take a Java course. I wasn't particularly interested in Java but nothing was particularly difficult about the class except the tests.

Not because I didn't understand the questions or logic flow, but it was administered in the way one might a test for a foreign language. I mean an actual spoken language, not [random programming language].

Specifically, we were given closed-book tests in which we were expected to hand-write code.

On paper.
With a pencil.
Scored on syntax.

To my mind, this wasn't testing the ability to think like a developer, but rather the ability to memorize syntax of the Java language. At the time AI didn't exist, and even auto-complete in the IDE wasn't a thing, and syntax does matter, but it felt disconnected from how software is actually built.

When it comes to writing code, specific domain knowledge of the syntax ranks fairly low on the scale of what's actually important. Developers constantly lean on other tools; Google, docs, books, and today AI.

If the job is software development, determining that the applicant can write code is worthwhile, but expecting them to sit down and write some arbitrary code without allowing reference to external sources especially under extra, artificial pressure of being watched within interview time constraints, is... unreasonable.

Speed is important.
Correctness is important.
Knowing and recognizing design patterns is important.

These all lead to stable, performant, maintainable code.

Call me an old man yelling at clouds if you like, but the job was never about writing code. It has always been about solving problems. Real developers use references constantly. Knowing when to reach for reference material, and where to look is much more important than memorization. Nobody ships in total isolation.

You could say that writing the code has been a bottleneck in the process. Modern tools -  autocomplete, LSPs, LLMs, etc - can reduce the bottleneck of typing, and shift focus to where it always belonged in the first place: Thinking.

AI doesn't replace engineers. It replaces the need to memorize. Even if you are not going to allow employees to use AI on the job, which frankly seems almost as nonsensical as trying to one-shot "vibe code" an application with AI, expecting someone to write functional code by hand, with no reference material is even more antiquated today than it was over two decades ago.

Instead, interviews should be a conversation, not a test. They should focus on how the candidate thinks about a problem. What they can determine from the specifications provided, what they would do about the portions that may be unclear or that they're unsure of.

Do they ask questions?
Do they think about edge cases?
Can they explain their approach to understanding and solving the problem?
If they can think of more than one way to approach it, can they articulate tradeoffs?
Can they adapt if requirements change?
Can they spot risks that might lead to stability or scalability issues?

I'm less interested in their ability to recall syntax, or regurgitate design patterns off the top of their head, than I am how they reason about a problem that needs to be solved.

Far too often interviews are disconnected from the reality of the job, with code challenges rewarding those who have recently interviewed, those who grind on "leetcode" or otherwise optimize for artificial constraints, completely overlooking judgement, thinking in systems, and most importantly real-world experience.

I've met and worked with incredible developers who suck at live coding, and mediocre ones who ace it.

Interviews should more closely resemble the job - not arbitrary code challenges or algorithm-puzzle style "trick question" nonsense. Most people never need to manually invert binary trees or implement their own sorting algorithms. These are solved problems that exist in most languages standard libraries or community code.

How candidates think about problems, communicate, and collaborate are vastly more important than if they can remember whether the haystack or needle comes first when searching an array or list, or immediately recall which design pattern applies to a specific problem. On the job, it's more often the case that the solution isn't actually a known target than a puzzle to unlock, but the latter is what is usually tested in interviews.

When I interview candidates for development roles I try to get a solid sense of how they think, and most importantly if they're someone I could see myself working with on a daily basis. For the most part, everything else can be learned.

In the age of AI-assisted development, this is even more relevant than ever, and interviews probably ought to adapt accordingly.

I see a lot of posts on X lately saying some version of: “The hardest part of building a SaaS isn’t the product, it’s the marketing.”

I agree with that statement, to a point. I also think it reveals something deeper, and a little uncomfortable.

The same problem shows up in job searches.

People talk about the job market as if it’s uniquely broken, uniquely hostile, or uniquely unfair right now. And yes, the market is challenging. There are real macro forces at play. But I don’t think finding a job is inherently harder than doing a job, any more than marketing a SaaS is inherently harder than building one.

They’re just different skill sets.

And many (dare I say most) people are very good at one, and absolutely terrible at the other.

Execution vs. marketing is a real divide

Developers are a great example.

Many engineers can implement complex systems with ease. They can design architectures, write clean code, ship features, debug production issues, and keep things running under pressure. Execution is their comfort zone.

But ask them to explain why what they built matters, to a customer, a hiring manager, or a non-technical stakeholder, and things often fall apart.

The value is there. The impact is there. The skill is there.

The story is missing.

The same thing happens in job searches.

You can be excellent at your job, consistently delivering, solving hard problems, making teams better, and still struggle mightily to land your next role. Not because you’re bad at what you do, but because you don’t know how to market yourself in a way that creates interest, confidence, and momentum.

Marketing isn’t lying even though people treat it like it is

A lot of technically strong people have an almost moral resistance to marketing.

• Marketing feels “salesy.”
• Marketing feels inauthentic.
• Marketing feels like exaggeration or spin.

So instead of learning how to do it well, they avoid it entirely.

But good marketing isn’t lying. It’s translation.

It’s taking something real and valuable and expressing it in a way that resonates with the audience you’re trying to reach.

In SaaS, that audience is customers.
In a job search, that audience is hiring managers, recruiters, and decision-makers.

In both cases, the failure mode is the same:
“I built something great. Why doesn’t anyone care?”

Being great at the work is not enough

This is the hard truth a lot of people don’t want to hear:

Being good at execution is table stakes.

It always has been.

Companies don’t hire potential value. They hire perceived value.
Customers don’t buy technical merit. They buy outcomes, clarity, and confidence.

If you can’t clearly articulate:

what you do,

why it matters,

and why you are the right person to do it,

then you’re relying on hope instead of strategy.

And hope is not a plan. Whether you’re launching a product or looking for a job.

The same pattern, over and over

When I zoom out, I see the same dichotomy everywhere:

• Builders who can ship but can’t sell
• Operators who deliver but can’t pitch
• Experts who assume the work should speak for itself

In reality, the work rarely speaks unless you teach it how.

That doesn’t mean everyone needs to become a marketer. But it does mean that if you want leverage (customers, jobs, opportunities) you need to develop at least a functional level of marketing skill.

Not hype.
Not bullshit.
Just clear, compelling communication.

The job search is a marketing problem

For many people, the job search isn’t failing because they’re unqualified.

It’s failing because they’re invisible, unclear, or indistinguishable.

Resumes list responsibilities instead of impact.
Interviews focus on tasks instead of outcomes.
Online profiles read like internal documentation instead of value propositions.

That’s not a talent problem. It’s a marketing gap.

And just like with SaaS, the people who close that gap, even imperfectly, tend to win disproportionately.

If you’re struggling to sell a product, or struggling to land a role, it might be worth asking a hard question:

“Am I actually bad at this… or am I just bad at explaining why it matters?”

In both SaaS and careers, execution gets you in the game.
Marketing is what moves the ball.

Ignoring that doesn’t make you principled.
It just makes you harder to find.

oklch is a great addition to css, providing much better control over color modifications, support for wider gamut, etc - reasons I'm not going to go into in this post.

But... sometimes you still might need, or want, good old hex colors. Email templates immediately come to mind.

So, if you happen to be collecting client branding colors in your application, for instance, and you're storing them as oklch instead of hex values, you'll need a way to convert to hex to inject the right value into your email template. Or whatever other use cases you can come up with.

If you're just doing one-off conversions, go ahead and visit oklch.com. But, if you need it dynamic, you might want some PHP code to do the conversion automatically for you.

I got you, fam:

<?php

/**
 * Convert OKLCH string to Hex
 * @param string $oklchString - OKLCH color string (e.g., "oklch(70% 0.15 30)" or "oklch(0.7 0.15 30deg)")
 * @return string Hex color string (e.g., "#ff5733")
 * @throws InvalidArgumentException
 */
function oklchToHex(string $oklchString): string
{
    // Parse OKLCH string
    if (!preg_match('/oklch\s*\(\s*([0-9.]+)%?\s+([0-9.]+)\s+([0-9.]+)(?:deg)?\s*\)/i', $oklchString, $match)) {
        throw new InvalidArgumentException('Invalid OKLCH string format');
    }
    
    $l = (float) $match[1];
    $c = (float) $match[2];
    $h = (float) $match[3];
    
    // Convert percentage lightness to 0-1 range
    if (str_contains($oklchString, '%')) {
        $l *= 0.01;
    }
    
    // Convert OKLCH to OKLAB
    $hRad = deg2rad($h);
    $a = $c * cos($hRad);
    $b = $c * sin($hRad);
    
    // Convert OKLAB to linear LMS (corrected matrix)
    $l_ = $l + 0.3963377774 * $a + 0.2158037573 * $b;
    $m_ = $l - 0.1055613458 * $a - 0.0638541728 * $b;
    $s_ = $l - 0.0894841775 * $a - 1.2914855480 * $b;
    
    $l3 = $l_ * $l_ * $l_;
    $m3 = $m_ * $m_ * $m_;
    $s3 = $s_ * $s_ * $s_;
    
    // Convert linear LMS to linear RGB (corrected matrix values)
    $r = 4.0767416621 * $l3 - 3.3077115913 * $m3 + 0.2309699292 * $s3;
    $g = -1.2684380046 * $l3 + 2.6097574011 * $m3 - 0.3413193965 * $s3;
    $b_rgb = -0.0041960863 * $l3 - 0.7034186147 * $m3 + 1.7076147010 * $s3;
    
    // Convert linear RGB to sRGB
    $r = linearToSrgb($r);
    $g = linearToSrgb($g);
    $b_rgb = linearToSrgb($b_rgb);
    
    // Clamp and convert to 8-bit
    $r = max(0, min(255, (int) round($r * 255)));
    $g = max(0, min(255, (int) round($g * 255)));
    $b_rgb = max(0, min(255, (int) round($b_rgb * 255)));
    
    // Convert to hex
    return sprintf('#%02x%02x%02x', $r, $g, $b_rgb);
}

/**
 * Convert linear RGB value to sRGB
 * @param float $val
 * @return float
 */
function linearToSrgb(float $val): float
{
    // Clamp to valid range first
    $val = max(0, min(1, $val));
    
    if ($val <= 0.0031308) {
        return 12.92 * $val;
    }
    return 1.055 * pow($val, 1 / 2.4) - 0.055;
}

// Example usage:
echo oklchToHex('oklch(70% 0.15 30)') . PHP_EOL;      // Warm orange
echo oklchToHex('oklch(0.5 0.2 200deg)') . PHP_EOL;   // Blue
echo oklchToHex('oklch(90% 0.05 120)') . PHP_EOL;     // Light green
echo oklchToHex('oklch(0.3 0.1 300deg)') . PHP_EOL;   // Dark purple

Prefer JS? No worries:

/**
 * Convert OKLCH string to Hex
 * @param {string} oklchString - OKLCH color string (e.g., "oklch(70% 0.15 30)" or "oklch(0.7 0.15 30deg)")
 * @returns {string} Hex color string (e.g., "#ff5733")
 */
function oklchToHex(oklchString) {
  // Parse OKLCH string
  const match = oklchString.match(/oklch\s*\(\s*([0-9.]+)%?\s+([0-9.]+)\s+([0-9.]+)(?:deg)?\s*\)/i);
  
  if (!match) {
    throw new Error('Invalid OKLCH string format');
  }
  
  let l = parseFloat(match[1]);
  let c = parseFloat(match[2]);
  let h = parseFloat(match[3]);
  
  // Convert percentage lightness to 0-1 range
  if (oklchString.includes('%')) {
    l = l / 100;
  }
  
  // Convert OKLCH to OKLAB
  const hRad = (h * Math.PI) / 180;
  const a = c * Math.cos(hRad);
  const b = c * Math.sin(hRad);
  
  // Convert OKLAB to linear LMS
  const l_ = l + 0.3963377774 * a + 0.2158037573 * b;
  const m_ = l - 0.1055613458 * a - 0.0638541728 * b;
  const s_ = l - 0.0894841775 * a - 1.2914855480 * b;
  
  const l3 = l_ * l_ * l_;
  const m3 = m_ * m_ * m_;
  const s3 = s_ * s_ * s_;
  
  // Convert linear LMS to linear RGB
  let r = 4.0767416621 * l3 - 3.3077115913 * m3 + 0.2309699292 * s3;
  let g = -1.2684380046 * l3 + 2.6097574011 * m3 - 0.3413193965 * s3;
  let b_rgb = -0.0041960863 * l3 - 0.7034186147 * m3 + 1.7076147010 * s3;
  
  // Convert linear RGB to sRGB
  r = linearToSrgb(r);
  g = linearToSrgb(g);
  b_rgb = linearToSrgb(b_rgb);
  
  // Clamp and convert to 8-bit
  r = Math.max(0, Math.min(255, Math.round(r * 255)));
  g = Math.max(0, Math.min(255, Math.round(g * 255)));
  b_rgb = Math.max(0, Math.min(255, Math.round(b_rgb * 255)));
  
  // Convert to hex
  return '#' + [r, g, b_rgb].map(x => x.toString(16).padStart(2, '0')).join('');
}

function linearToSrgb(val) {
  // Clamp to valid range first
  val = Math.max(0, Math.min(1, val));
  
  if (val <= 0.0031308) {
    return 12.92 * val;
  }
  return 1.055 * Math.pow(val, 1 / 2.4) - 0.055;
}

// Example usage:
console.log(oklchToHex('oklch(70% 0.15 30)'));      // Warm orange
console.log(oklchToHex('oklch(0.5 0.2 200deg)'));   // Blue
console.log(oklchToHex('oklch(90% 0.05 120)'));     // Light green
console.log(oklchToHex('oklch(0.3 0.1 300deg)'));   // Dark purple

Happy color converting!

FrankenPHP is pretty badass, especially when used with Laravel Octane to make your Laravel app really fly. But what about real-time feedback to the frontend?

As it happens, FrankenPHP ships with Mercure support out of the box. But configuration can be a little challenging. Having just fought with it for longer than I'd like to admit, I thought I'd share some of the useful tidbits I learned, in hopes of saving someone else the brain damage.

For my purposes, I'm running the ServerSideUp FrankenPHP Docker image, the first challenge was deciphering just how to go about getting it setup at all because the docs there only mention that it's available, but not how to configure.

At first I just tried setting environment variables in my docker config, which seemed to .. sort of work, but I just continually received 403 errors. Then I decided to try the CADDY_SERVER_EXTRA_DIRECTIVES env var, and define the mercure module config:

CADDY_SERVER_EXTRA_DIRECTIVES: |
  mercure {
    publisher_jwt !mySuperSecretKey!
    anonymous
  }

After that change, I was getting a 401 response.. progress, but still no joy.

I'll be brief; the jwt secret needs to be surrounded by ! if it's a hash (base64, for instance) but if it's hex, you need to omit the ! or it'll misinterpret the key and your properly signed request will be treated as invalid.

So, tl;dr which doesn't seem to be documented very well anywhere:

openssl rand -base64 48

Produces something like:
7/SrS1rBcVyw2NK2sj2VD2guaK6PNlfnIDIHse6mBpWE7N2jTt1pXldMnKsmA

Surround it with !

openssl rand -hex 48

Produces something like:
483b2395ceaee186816bfa6972d4a8a8d7025c838220c4fe2e3dfa43e907ee7

Do NOT surround it with !

The other issue I ran into was that it complained that it could not open the boltDB for writing, that was easily solved by explicitly defining the transport path to a writeable directory, in my case one I mounted from my host so it persists if I remove the container, and more important so I could just easily see that it created the file... 😅

My final env entry to get it working looks like this (not my actual keys):

CADDY_SERVER_EXTRA_DIRECTIVES: |
  # Mercure Hub for real-time Server-Sent Events
  mercure {
    # Publisher JWT secreA
    publisher_jwt 123abc HS256

    # Subscriber JWT Secret
    subscriber_jwt 456xyz HS256

    # Allow anonymous listeners
    anonymous

    # Allow CORS from anywhere
    cors_origins *

    # Allow Publishing from anywhere (or maybe just your server)
    publish_origins *

    # Tell it where to write the bolt db, somewhere it CAN write
    transport bolt {
      path /opt/frankenphp/mercure.db
    }
  }

All the available mercure directives can be used here.

So, why go to the trouble? After all I could just use Reverb, right? Well websockets are inherently bi-directional, and I have no need for that, where as mercure is really intended for sending to client-side subscribers, and for mobile users, is easier on the battery/etc. It's a pretty cool protocol that I anticipate using more often moving forward. And, bonus - no need for another docker container or hoop jumping to run Reverb in my frankenphp container, which means I have a few more resources to spread around the other services, and one less container to orchestrate and manage.

I've been using Pagefind quite happily on my JAMStack sites for quite some time, but one thing that's always bothered me a little bit is the default behavior when moving through browser history.

By default, when you run a search and then click a result all works as you'd expect, but when you click Back in my experience the last entered search term remains populated in the search field, but no search results are displayed - obviously because those get populated dynamically via javascript, and I assume some browser state is keeping the input field filled.

You can force it to run a search if there's a search string in the URL query parameters, I think a lot of people have discovered and implemented that, and it's great especially if you're adding the schema json to make search box available to google (whether they ever choose to enable it for your site or not)

But wouldn't it be great, if instead of a populated search field with no results, or it rerunning a potentially incorrect prior search when you navigate back, if it instead always reran your last search phrase? Whether that was from clicking a link with the search term baked in, using the google search box, or having manually entered a search phrase.

It would be great.. no, it IS great! Just a handful of JS makes it work.. we simply need to add some listeners and update the URL every time it changes, and then along with watching for a value populated to q in the URL, we also need to listen for popstate events to trigger search on back-button navigation.

Here's a bit of code that makes it go:

<script>
  // This assumes you have your baseUrl defined in some site data
  // and that your search page lives at /search/
  // and that your search phrase is defined with the "q" GET parameter
  
  const BASE_URL = '{{ site.baseUrl }}/search/';

  const makeUrl = term =>
    term ? `${BASE_URL}?q=${encodeURIComponent(term)}` : BASE_URL;

  const debounce = (fn, ms = 250) => {
    let t;
    return (...args) => {
      clearTimeout(t);
      t = setTimeout(() => fn(...args), ms);
    };
  };

  window.addEventListener('DOMContentLoaded', () => {
    const pagefind = new PagefindUI({
      autofocus: true,
      element: '#search',
      showSubResults: true,
    });

    const params = new URLSearchParams(location.search);
    const initial = params.get('q') || '';
    if (initial) pagefind.triggerSearch(initial);

    const input = document.querySelector('input.pagefind-ui__search-input');
    const clearButton = document.querySelector('button.pagefind-ui__search-clear')

    input.addEventListener('input', debounce(e => {
      const term = e.target.value.trim();
      history.replaceState({ q: term }, '', makeUrl(term));
    }));

    clearButton.addEventListener('click', debounce(e => {
      history.replaceState({}, '', makeUrl(null));
    }));

    window.addEventListener('popstate', e => {
      const term =
        (e.state && e.state.q) ||
        new URLSearchParams(location.search).get('q') ||
        '';

      input.value = term; 
      if (term) pagefind.triggerSearch(term);
      else clearButton.click();
    });
  });
</script>

I'm beginning to wonder what that actual point of fat arrow functions in PHP really are. With the exception of multi-line logic, I figured they basically worked the same way they do in JavaScript, giving you full, direct access to the parent scope. But, that is clearly not how they work!

In JavaScript if you use the arrow function syntax you get direct access to the outer scope - SUPER handy! So in JS I might do something like this:

const foo = [1,2,3,4]
const bar = ['one', 'two', 'three', 'four']

const baz = bar.flatMap((i) => [foo.shift(), i])

// baz === [1, 'one', 2, 'two', 3, 'three', 4, 'four']

With the arrow function syntax I can access foo an it works as expected, giving me the first item from the array on each iteration.

In PHP, if I try the same:

$foo = [1,2,3,4];
$bar = ['one', 'two', 'three', 'four'];

// Using Laravel's collect for simplicity here:
$baz = collect($bar)
          ->flatMap(fn ($i) => [array_shift($foo), $i])
          ->toArray();

// $baz === [1, 'one', 1, 'two', 1, 'three', 1, 'four']

PHP gives me access to the outer scope with a fat arrow function, so I don't have to explicitly pass variables into my closure, which is nice, but.. they don't work the way I expect. I'm sure there's a good PHP internals reason for this, and the best I can figure is that fat arrows are just syntactic sugar for a function that simply returns whatever is after the arrow, and makes all the things available as if you manually passed everything in a use (), but it clearly does so by COPY not by REFERENCE...

Because even if I use the more verbose closure syntax:

$foo = [1,2,3,4];
$bar = ['one', 'two', 'three', 'four'];

// Using Laravel's collect for simplicity here:
$baz = collect($bar)
          ->flatMap(function ($i) use ($foo) {
              return [array_shift($foo), $i];
          })
          ->toArray();

// $baz === [1, 'one', 1, 'two', 1, 'three', 1, 'four']

I get the same thing. I have to specifically use the more verbose closure syntax and explicitly pass my $foo array into the closure by reference:

$foo = [1,2,3,4];
$bar = ['one', 'two', 'three', 'four'];

// Using Laravel's collect for simplicity here:
$baz = collect($bar)
          ->flatMap(function ($i) use (&$foo) {
              return [array_shift($foo), $i];
          })
          ->toArray();

// $baz === [1, 'one', 2, 'two', 3, 'three', 4, 'four']

This again, kind of feels unnecessary, because array_shift() explicitly accepts the array you pass into it by reference ... so one, like myself, might logically conclude that it should behave the way the JS version does.

Best I can figure is that in a closure, whatever you pass in, is passed by COPY on every iteration, unless you've explicitly passed it in by REFERENCE – and there's no way to DO THAT in a fat arrow function. 🤬

This leaves me wondering, truly, wtf is the actual point of fat arrow functions in PHP? They don't really seem to give that much value, other than slightly less verbosity - which, granted, is often a good thing.. but the lack of being able to do multiple lines, a la:

let foo = (bar) => {
  //do some stuff

  // do some other stuff

  return 'something'
};

...and apparently only giving restrictive access to the parent scope... what's the point?

Let's start by addressing the elephant in the room – ideally one should not be posting thousands of rows in a request. They should instead send many smaller batches, or upload a file somewhere to process in chunks with a queue.

But, let's be real - we don't live or work in an ideal world, and sometimes you just need the unreasonably large http request of data to work, and be validated in a reasonable amount of time (what that amount is, is debatable).

I love Laravel, I have always loved Laravel. I'm in the midst of migrating a large API from a JS backend to Laravel, changing as little as possible about the api in the process because the application that consumes it cannot be rebuilt at this time. Thus, queues and the like for processing the data posted to the endpoint in question are off the table.

Now, there is already validation happening client-side on this data, so the naive approach might be to say that's sufficient and just assume good data has been submit, but it's never a good plan to trust user input, so that's off the table too – and, I won't comment on whether or not that was how it used to be... 😉

What we're working with is a bit of general info for a top-level object, and then an array of items consisting mostly of a location name and address parts. These records will be attached as children to the top level object, for someone to work through and manually process later within the application.

For the most part, our validation rules are little more than required or nullable and string though there are also a couple date fields to set a start & end date, that have to adhere to rules around their expected format, ensuring the start comes before the end, and that both dates fit within an allowable window. But still nothing too crazy.

Here's the validation rules I started with in my form request – formatted for clarity & string notation instead of arrays for brevity:

return [
  'name'                   => 'required',
  'organization_id'        => 'required|exists:organizations,id',
  'tags'                   => 'array',
  'tags.*.name'            => 'required',
  'items'                  => 'required|array|min:1|max:5000',
  'items.*.name'           => 'required|string',
  'items.*.notes'          => 'nullable|string',
  'items.*.start_date'     => 'required|date:Y-m-d|before_or_equal:items.*.end_date|'.
                              'before_or_equal:'.$soonest.'|after_or_equal:'.$oldest,
  'items.*.end_date'       => 'required|date:Y-m-d|after_or_equal:items.*.start_date|'.
                              'before_or_equal:'.$soonest.'|after_or_equal:'.$oldest,
  'items.*.street_address' => 'required|string',
  'items.*.city'           => 'required|string',
  'items.*.state'          => 'required|string',
  'items.*.zip'            => 'required|string|min:5|max:10',
  'items.*.tags'           => 'array',
  'items.*.tags.*'         => 'string',
];

Of course my first thought was that maybe the date logic was slowing things down, so I tried to remove it first.. but no joy - on small datasets it does alright, but once you start approaching the max allowed items of 5,000 it will consistently timeout. If I manually override the max execution time

// Uncap the execution time, or even just set a large duration
ini_set('max_execution_time', 0);

it will complete, but in over a minute on my new M4 Pro mac mini, several minutes on my M2 macbook air.. too slow in any case. Especially when if we simply omit all of the per-row item validation items.*.whatever and simply ensure that items is required, and is an array of appropriate size, the full request lifecycle is only about 250ms from the time I submit a request with 5,000 items in the payload, until it creates all those records, and returns a successful response.

Yikes!

250ish milliseconds, vs – best case – 1 minute.

But, as we've already covered, we obviously don't want to just assume the user input is valid and safe. Even though we're doing client side validation in the application that talks to this api. Eventually we may have outside consumption of the api, or someone could manually post to it – assuming they have/can find proper authentication and whatnot. In any case.. we need to solve this.

There are several issues on github and elsewhere referencing this issue (that's been around for many years), wherein validation with the asterisk to apply rules to array items is sloooow... and while PRs are welcome, sadly the position of the core team thus far has simply been along the lines of "well, you shouldn't be doing that sort of thing." Which, feels like the wrong response when we're talking about a framework that is all about developer experience, etc. But that's a potentially ranty tangent we need not go down.

The good news is - I've found a solution that is gonna work for me!

Thanks to the good folks at Spatie, we have Laravel-Data available, which can be used in place of Form Request Objects, and it supports validation of nested items. So naturally that would be one's first inclination (or maybe just mine) as the next thing to try.

So I replaced my FormRequest Object with a LaravelData object, added my first chunk of permissions, then created another data object for the items in my items array, and applied the relevant permissions to that.

Defining items as an array, and including detail in the PhpDoc block that it should be an array of my Item Data objects got validation working, and there was indeed some performance improvement, but not enough. It came down from about a minute to 35-45 seconds.

We'll call it maybe a 40% improvement. That's not bad, but I wondered whether I could do better because that was still pretty snail-like, and after all I was still looking at the unvalidated speed of 250ms and we're still way too far off of that mark to quit.

So, I reverted back to my prior Form Request object, which I already knew worked well if I only validated the top level stuff and basic validation that items was a required array of 1 - 5000 rows long.

Leaving my validation as just that in the form request object, I added one single line to the start of the method these requests gets routed to:

$items = collect($request->validated('items'))
  ->map(
    fn($item) => ItemData::factory()
                   ->alwaysValidate()
                   ->from($item)
  );

So I'm grabbing a collection from the validated items – we know it's an array of appropriate size – then mapping that into instances, ensuring the validation rules are applied, for each individual item. Instead of it applying array validation to the whole set.

The end result is that the same set of 5,000 records that previously took a minute or more, very probably simply timing out the request in most cases – even if you actually have the ability to override max execution time – now completes in about 2 seconds.

We're still not winning any major speed awards, but we do have validated data, and appropriate validation error message responses... did I forget to mention, if any of those objects fail to instantiate it'll throw a 422 response with validation errors? Anyway, we're validating the user input, and doing so in, what I'll consider a reasonable response time given that, as discussed at the beginning, this is definitely a sub-optimal way to ingest this data.

Also of note, in practice the users uploading this data are only sending at most a couple hundred records at a time, so for them this should be plenty fast. (100 completes in about 120ms)

So, here's where we ended up. Rules in the FormRequest object now look like this:

return [
  'name'            => 'required',
  'organization_id' => 'required|exists:organizations,id',
  'tags'            => 'array',
  'tags.*.name'     => 'required',
  'items'           => 'required|array|min:1|max:5000',
];

I have the collection map from above as the first line in my Invokable class that the endpoint is mapped to, and I have this Data object to handle validation of the individual items:

class ValidOrderItemData extends Data
{
    public function __construct(
        public string $name,
        public ?string $notes,
        public string $start_date,
        public string $end_date,
        public string $street_address,
        public string $city,
        public string $state,
        public string $zip,
        public ?array $tags,
    ) {}

    public static function rules(): array
    {
        $soonest = now()->format('Y-m-d');
        $oldest = now()->subYear()->startOfMonth()->format('Y-m-d');

        return [
            'name' => ['required', 'string'],
            'notes' => ['nullable', 'string'],
            'start_date' => ['required', 'date:Y-m-d', 'before_or_equal:end_date', "before_or_equal:$soonest", "after_or_equal:$oldest"],
            'end_date' => ['required', 'date:Y-m-d', 'after_or_equal:start_date', "before_or_equal:$soonest", "after_or_equal:$oldest"],
            'street_address' => ['required', 'string'],
            'city' => ['required', 'string'],
            'state' => ['required', 'string'],
            'zip' => ['required', 'string', 'min:5', 'max:10'],
            'tags' => ['array'],
            'tags.*' => ['string'],
        ];
    }
}

...and just like that, I'm validating a large array of data about 30x faster than using the wildcard property name validation rules will provide out of the box, and at the end of the day it's not really all that much more effort.

I've happily been using Netlify for several years, and was surprised to learn randomly, around 2020/2021 or so, that cloudflare quietly put out a competing product. My initial tests of cloudflare pages back when it was new were disappointing with excrutiatingly slow build times, but boy has that ever changed!

Cloudflare has definitely made big strides in boosting performance of builds to their pages deployments, and it's now just as efficient and quick as (maybe even quicker than) Netlify. So the time to jump was nigh.

But, if I was happy with Netlify, why change?  Well the main reason was that cloudflare pages are served up with http/3 rather than just http/2. While that isn't a huge deal at the moment, I didn't feel like being stuck in the (recent) past as Netlify has shown no interest in upgrading.

Migration, ultimately, was very straightforward. I created a new pages project over at cloudflare pointed at my github repo

I then added a custom domain to this new pages setup:

This required me to update my DNS to change the existing CNAME record from Netlify to point to cloudflare.

I already had redirects setup on my Netlify deployment, by publishing them to a _redirects file, in the format:

/old/path1		/new/redirected/path1		308
/old/path2		/new/redirected/path1		302
/old/path3		/temp/redirection			301

Previous location, new location, redirect code. Pretty simple... and cloudflare pages support the exact same format in a file with the same name. Doesn't get any easier than that!

The only notable change I had to make was to add a new _headers file, because that previously was defined in a toml file for the netlify deployment.

My old netlify headers were defined thusly:

[[headers]]
  for = "/*"
  [headers.values]
    Strict-Transport-Security = "max-age=63072000; includeSubDomains; preload"

The new header is defined a little more succinctly in a _headers file as noted...

/*
  Strict-Transport-Security: max-age=63072000; includeSubDomains; preload

Basically just the applicable path to apply the header(s) to, and then the headers on indented lines following that. Pretty simple stuff.

I did have a function in place on netlify to push sitemap updates to google on deploy, but google has deprecated that endpoint so I didn't migrate the post-deploy function to cloudflare, but yep - it totally supports those too.

One of the other niceties of cloudflare pages is that it has built in analytics, which if all you care about is seeing core web vitals and basic visits, it could be a viable, lighter weight, replacement for google analytics:

For just basic visits, by page & country/etc, this is totally serviceable data and you may not need anything beyond this. It's really incredible that this is all free!

The last bit of config I did was to ensure that my site is only accessible at the custom domain, not the .pages.dev domain – other than preview deploys, which, oh yes are also included, and better can be locked down to only be accessible to people you specify should be allowed to access them!

I left the netlify deployment in place to ensure that I wouldn't break things for anyone in case the DNS hadn't propagated, but now that it's been humming along nicely on cloudflare for over a week, I should probably get around to removing that.

In all, this was easily the most painless migration I've ever done.

In case you're curious - the site in question is the site for my voiceover business, which is built with the static sitebuilder, Eleventy, Tailwind CSS and a sprinkle of AlpineJS. I'm very pleased with how easy it is to maintain and extend, and especially with how quickly it builds and even more so how fast it's served.

If you're in the market for any voice work, be it for a video game, phone messaging, training materials for an app, or anything else – I'd love to hear from you! Check out the site over here at willvincentvoice.com.

Update:

So there was one small issue I had to sort out when I shut off Netlify. Because cloudflare pages don't allow you to setup an A record, unless you let cloudflare manage the domain anyway, and Netlify did, I needed to setup a redirect from the non-www version of my domain to the www version that is setup as the custom domain for the cloudflare pages hosted site.

Conveniently, I'm already running a bunch of miscellaneous services with docker on my server, and I use traefik to shuttle traffic around between those various docker instances. So setting up a redirect was basically just a matter of configuring a simple container. While in theory I should be able to do it with just traefik, I couldn't get it to work and ended up using the morbz/docker-web-redirect docker container setting the env var VIRTUAL_HOST to my non-www domain, and the REDIRECT_TARGET to the www version. I also included relevant traefik config to rewrite http to https, etc.

So, one small wrinkle if you're not letting cloudflare manage your domain I guess.

There's no denying that Amazon AWS's S3 product is pretty fantastic and very flexible. It's especially interesting when you leverage S3 Select to query your CSV, JSON, or Parquet files.

If you're not familiar, Amazon S3 Select allows you to query your files for data using a simplified SQL syntax. It is SQL syntax but simplified because it only supports a fairly small subset of the SQL language. But that's ok, there's still a LOT that can be done:

• Count queries, to find the number of matching results or number of lines in a file.
• Simple WHERE clauses, to grab a subset of data, to populate a chart on a website for instance.
• Pagination... sort of. This is what this article is about.

Suppose you have a costly query that takes your primary data source minutes or more to run, you might want to cache those results so that it's less painful to use elsewhere. This might be a good case for something like Mongo, DynamoDB, etc. but S3 is also a perfectly viable option.

Say you output a report as a CSV, and store it in S3 to later be displayed as tabular data on a website. That's pretty straightforward. But suppose the file contains hundreds, thousands, or millions of rows. You surely wouldn't want to try to display a file that large in the browser - any sane person would want to paginate that data

S3 Select provides us with the LIMIT clause to specify how many rows we want to select, which gets us halfway there. Anyone who's manually implemented pagination queries knows that it's a function of LIMIT and OFFSET clauses to grab a window of desired size from somewhere within the overall result set.

Unfortunately, S3 Select does not offer the OFFSET clause. ☹️

As it turns out though, the solution isn't too terrible. As long as you're able to inject a row counter or serialized id into your report data when you store it in S3 for later consumption, you can easily replace the OFFSET clause with a WHERE clause:

SELECT * FROM s3Object LIMIT 5 OFFSET 10

(which does not work)

is functionally the same as

SELECT * FROM s3Object WHERE row > 10 LIMIT 5

(which does work)

I've recently implemented this solution for reporting activities in one of the apps I'm responsible for at my day job because certain reporting was taking so long to run that things were timing out for folks, so we've instead moved to generate those asynchronously, and firing a callback when the report completes.

This allows reports that previously wouldn't finish to not only finish but for those that did but were still slow to perform more efficiently when flipping through pages, as otherwise, previously the on-demand requests were run for every pagination page-turn. So if page 1 took say 30 seconds to respond, so did page 2, etc. However, with the asynchronous generation the whole report might take a minute or two to run, but then pagination is speedy as it's just fetching a subset of already prepared rows from a file on S3.

It probably would still be even more performant to populate the report data into a database, but since people download the CSV file, we'd end up generating it or streaming the data in that format at some point anyway so the little bit of performance we give up is an acceptable trade-off.

You could also probably accomplish this with the ScanRange parameter, providing start and end bytes to define a range of the file to scan. But given that it's based on byte sizes rather than row counts, I'm not sure how you could reliably paginate equal-sized pages of data this way. That would probably be better for processing chunks of data, rather than for display.

Anyway, now you know how to work around the lack of the OFFSET clause with S3 Select. Happy coding!

I recently had a need to populate the available options for a multiselect component with values from a remote API. ChoicesJS supports this, though it's documentation leaves much to be desired. As such, a new post detailing how I made my dynamic multiselect work seemed important.

What exactly do I mean by "dynamic multiselect" you ask? Simply that, it's a select component allowing multiple choices, whose options are fetched - dynamically - from somewhere else, not preloaded up front.

The scenario here is that my users need to be able to select one or more resources, but they may have hundreds or thousands in the list of available resources, so loading all the options up front to populate the list of selectable options was out of the question. That would have greatly simplified things of course.

So, what is the desired functionality, and how do you build it?

I'm glad you asked. The desired behavior is that when a user enters their search term into the input field ChoicesJS presents us, we want to trigger an API call that will fetch the matched resources and populate our list of available options, allowing them to make one or more selections. Then, they can optionally search again, selecting more options, and so forth.

Since we're using Livewire, all of the api interaction is going to happen server side, the beauty of this is that no credentials need to be exposed to the client side, and any heavy data manipulation will also occur server side.

Ok, first things first, we need a livewire component

<?php

namespace App\Http\Livewire;

use Livewire\Component;
use MyApi\Client;

class Select extends Component
{
  public $options = [];
  public $selections = [];

  public function render()
  {
    return view('livewire.select');
  }
}

That's the basics done, but we also need to write a method to call when the user executes a search:

public search($term)
{
  $results = Client::search($term);

  $preserve = collect($this->options)
                ->filter(fn ($option) =>
                    in_array(
                      $option['value'],
                      $this->selections
                    )
                  )
                ->unique();

  $this->options = collect($results)
                     ->map(fn ($item) => 
                             [
                               'label' => $item->name,
                               'value' => $item->id
                             ])
                     ->merge($preserve)
                     ->unique();

  $this->emit('select-options-updated', $this->options);
}

What's happening here?

1. We pass the entered search phrase to the search functionality of our api client. This could also be something as basic as a whereLike query against a laravel model.
2. We look at the existing options, and pluck out any that are currently selected.
3. We map the relevant data from the search results into the format expected by our ChoicesJS widget, and then merge in the options that we grabbed in the previous step.
4. Emit an event so that the JS widget can update its options list.

Pretty straightforward... but why step 2, and merging results of it into the search query results?

Well, because the way we have choices wired up, it isn't aware of choices that aren't presently in the available options list. You could probably store the entire object in there if you really wanted to, but then you'd be hauling around a lot more data on subsequent request/response calls... and this is pretty easy - and neat.

That's the PHP side sorted, now on to the view

@props([
  'options' => [],
])

@once
  @push('css')
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/choices.js/public/assets/styles/choices.min.css" />
  @endpush
  @push('js')
    <script src="https://cdn.jsdelivr.net/npm/choices.js/public/assets/scripts/choices.min.js"></script>
  @endpush
@endonce

<div x-data="{}" x-init="">
  <select x-ref="select"></select>
</div>

There's our basic view framework stubbed out. Notice the use of @once, and @push to push the requisite css & js into relevant stacks, and ensure they're only added one time, so if you had multiple instances of this component on you page you wouldn't be polluting the DOM with several copies of the same script. Glorious. It's all the little niceties like this that make me love Laravel...

Ok, so we're loading the choices script and styles, now we need to point them at our lonely little select element and make it go. One piece at a time here..

Lets flesh out the x-data

x-data="{
  value: @entangle('selections'),
  options: {{ json_encode($options) }},
  debounce: null,
}"

• The value gets entangled, linked with, the livewire state.
• options get populated with any pre-defined values we may have set by default in our php component
• debounce will be used as a target to track a setTimeout() to debounce user input, you don't really need to define it, but for clarity I've included it.

Now the meat of the implementation, the x-init

x-init="this.$nextTick(() => {
  const choices = new Choices(this.$refs.select, {
    removeItems: true,
    removeItemButton: true,
    duplicateItemsAllowed: false,
  })

  const refreshChoices = () => {
    const selection = this.value

    choices.clearStore()

    choices.setChoices(this.options.map(({ value, label }) => ({
      value,
      label,
      selected: selection.includes(value),
    })))
  }


  this.$refs.select.addEventListener('change', () => {
    this.value = choices.getValue(true)
  })

  this.$refs.select.addEventListener('search', async (e) => {
    if (e.detail.value) {
      clearTimeout(this.debounce)
      this.debounce = setTimeout(() => {
        $wire.call('search', e.detail.value)
      }, 300)
    }
  })

  $wire.on('select-options-updated', (options) => {
    this.options = options
  })

  this.$watch('value', () => refreshChoices())
  this.$watch('options', () => refreshChoices())

  refreshChoices()
})"

Alright there's a bunch going on there, but nothing to terribly complicated.

1. We instantiate an instance of ChoicesJS, pointing at the select element we previously created by it's x-ref value 'select', with some basic config.
2. Create a function that will refresh the available options whenever we call it, iterating through options and setting selected to true for any items whose value is in our selections wire model bucket.
3. Set up an event listener to sync the selections whenever a change event fires.
4. Set up an event listener to call our PHP search method, debounced with a 300ms timeout.
5. Set up a listener for the livewire event that we emit when we've updated the options on the server side, and accordingly update them on the client side.
6. Set up watchers that call our refresh function when the selections or options have changed.
7. Finally, fire the refresh function to setup the initial state of the choicesJS widget

This is all wrapped in a $nextTick because the markup and JS libraries/etc need to be on the page before it will work.

I can't take credit for all of this, Caleb nicely outlined how to work with the choicesJS library along with alpine in an integrations write up on the alpineJS site. But the search and $wire.on bits are all me.

So that's it. When you type into the choicesJS widget, it'll hit the api client and fetch results, those results will populate into the choices list. When you make a selection it'll work the same is if it'd been part of a pre-populated list all along. When you run a new search, previously selected elements will be merged with new search results so that the prior selects stay visible in the choicesJS widget.

You probably noticed there's no wire:model defined on the select element... well, that's because we're managing the state with alpine, so it's not necessary. choicesJS is going to override that select element anyway.

Our final livewire component & view

Livewire PHP Component

<?php

namespace App\Http\Livewire;

use Livewire\Component;
use MyApi\Client;

class Select extends Component
{
  public $options = [];
  public $selections = [];

  public function render()
  {
    return view('livewire.select');
  }

  public search($term)
  {
    $results = Client::search($term);

    $preserve = collect($this->options)
                  ->filter(fn ($option) => 
                    in_array(
                      $option['value'],
                      $this->selections
                    )
                  )
                  ->unique();

    $this->options = collect($results)
                       ->map(fn ($item) =>
                         [
                           'label' => $item->name,
                           'value' => $item->id
                         ])
                       ->merge($preserve)
                       ->unique();

    $this->emit('select-options-updated', $this->options);
  }
}

Blade View

@props([
 'options' => [],
])

@once
  @push('css')
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/choices.js/public/assets/styles/choices.min.css" />
  @endpush
  @push('js')
    <script src="https://cdn.jsdelivr.net/npm/choices.js/public/assets/scripts/choices.min.js"></script>
  @endpush
@endonce


<div
  wire:ignore

  x-data="{
    value: @entangle('selections'),
    options: {{ json_encode($options) }},
    debounce: null,
  }"

  x-init="
    this.$nextTick(() => {
      const choices = new Choices(this.$refs.select, {
        removeItems: true,
        removeItemButton: true,
        duplicateItemsAllowed: false,
     })

     const refreshChoices = () => {
       const selection = this.value
  
       choices.clearStore()

       choices.setChoices(this.options.map(({ value, label }) => ({
         value,
         label,
         selected: selection.includes(value),
       })))
     }

     this.$refs.select.addEventListener('change', () => {
       this.value = choices.getValue(true)
     })

     this.$refs.select.addEventListener('search', async (e) => {
       if (e.detail.value) {
         clearTimeout(this.debounce)
         this.debounce = setTimeout(() => {
           $wire.call('search', e.detail.value)
         }, 300)
       }
     })

     $wire.on('select-options-updated', (options) => {
       this.options = options
     })

     this.$watch('value', () => refreshChoices())
     this.$watch('options', () => refreshChoices())

     refreshChoices()
   })">

  <select x-ref="select"></select>

</div>

Final Thoughts

This component could, and probably should, be cleaned up to be more reusable. Making the event listened to definable as a prop passed into the blade component and so forth, but this should give you enough of an overview to get something like it working on your project.

As always, huge thanks to the community that makes any of this possible. Laravel, Livewire, AlpineJS and Tailwind truly make development a dream - I am a huge huge fan of the TALL stack!

Static site builders are pretty great, but how do you go about setting up redirects when you inevitably move or replace a page?

Thankfully, Netlify expects that you will eventually need to setup 301 and 302 redirects, and even allows you to redirect with a 404 or 200 response code too. Neat!

Their docs cover the format of the file pretty clearly, so I won't spend too much time rehashing that here, but suffice it to say, you need to have a file named _redirects in the docroot, with one redirect per line. Obviously you could manually create that file and have it get passthru copied during the build of your static site, but who wants to manually maintain such a file when we can let the sitebuilder handle it for us?

I use 11ty, not on this site, but for my voice over site (oh yeah, did you know I'm also a voice over actor?) - which is what we'll be discussing here. I imagine the process would be pretty similar for other sitebuilders but this is what I know so this is what we're covering... 😎

I use nunjucks, liquid is basically the same syntax, so either way, create a file in your src directory called _redirects.njk (or .liquid as applicable).

Within that file we need a little bit of front matter:

---
permalink: /_redirects
eleventyExcludeFromCollections: true
---

This tells 11ty to keep it out of the collections.all page collection, and to ensure it gets written to the file _redirects in the docroot when the site is built.

Pretty easy so far, but that won't do anything other that create a blank file.

Ideally, we want to be able to just specify the old path in the front matter where the thing lives now or of the new page that replaced that old thing.

Say something like: redirectFrom: /some/old/url

Or maybe we even want to be able to optionally redirect a bunch of old pages to this one new page, like we combined a bunch of things into one consolidated page - probably not the best move for SEO, but hey, you do you.

In that case we might just want to do something like this:
redirectFrom: ['/some/old/url', '/some/other/old/url', '/yet/another/url']

Alrighty then - lets get to it!

Lets think through the steps logically, we need to iterate through every page in our site, and check if it has a redirectFrom directive in its front matter, then determine if that's one url or an array of urls, and write out one redirect per line. For kicks, we'll also optionally support a redirectCode property.

First, lets open up the 11ty config file and add a new filter to determine if something is a string or not, this is how we'll check if our redirectFrom property is a string or an array - technically we probably ought to also check if it's an array, but we'll just assume it is if not a string since, we are the ones writing these files, after all...

Right. An is_string filter, looks a little something like this:

// file: .eleventy.js

module.exports = (config) => {
  // ..snip..
  
  config.addFilter('is_string', function(obj) {
    return typeof obj == 'string'
  })
  
  // ..snip..
}

Usage is simple, {% if thing | is_string %} and so forth.

Alright, that's all the bits in place, now we can populate the rest of our _redirects.njk file:

---
permalink: /_redirects
eleventyExcludeFromCollections: true
---
{%- for page in collections.all -%}
  {%- if page.url and page.data.redirectFrom -%}
    {%- if page.data.redirectFrom | is_string -%}
      {{ page.data.redirectFrom }}  {{ page.url }}  {{ page.data.redirectCode or '301'}}
    {%- else -%}
      {%- for oldUrl in page.data.redirectFrom -%}
        {{ oldUrl }}  {{ page.url }}  {{ page.data.redirectCode or '301'}}
        {%- if not loop.last -%}
          {{ '\n' }}
        {%- endif -%}
      {%- endfor -%}
    {%- endif -%}
    {{ '\n' }}
  {%- endif -%}
{%- endfor -%}

So this will look at every page, and assuming that page has a url AND a redirectFrom, it'll check if the redirectFrom is a string and if so spit out the old url, new url and optional redirect code (defaulting to 301).  If it's an array it'll do the same for each item in the array. Ensuring newlines are added after every individual line, without extra newlines (though that wouldn't hurt anything)

This will get rudimentary redirects working for your 11ty site hosted with Netlify straight away. What it does not cover is more complex redirects with query strings, params, and so forth.

Refer to the Netlify docs for details on the proper syntax for that. Config of that is beyond the scope of this blog post. While you're over there looking at the docs, take note you can also do fun proxy things by setting up redirects with 200 codes. Neat.

I don't know why, it was working, I didn't change anything significant on my machine, it just stopped working. Weird part is that it still works on another machine!

So, something about how the watcher works (or doesn't in my case) - it's failing to detect filesystem events, apparently. Even more strangely, the watch-poll option doesn't work either.

When I run yarn watch or yarn watch-poll it'll compile once, immediately, and then sit there pretending to watch for changes, but never compile anything again.

SUPER annoying, and super frustrating. I lost a whole day chasing this, and ended up without any more insight than I started with.

What I did, eventually, end up with though - is a workaround of my own.

I have a .scripts directory in the root of my project for various dev scripts, namely a release script that updates my changelog file automatically from all the commits since the last release, thanks to Standard Version for making that painless (why oh why is it deprecated?!) - and also a hotfix script that will cherry pick specified commit(s) and push only those to the production branch.

My deployments are automatic thanks to github actions & laravel vapor, so everything really is quite a breeze working on this project... or was, until this issue turned up.

Anyway, the workaround. I'm sure I'm not the only person to have to fight with this, as I know there are issues with the way filesystem events on macOS work.. (or, again, don't).

In my .scripts directory I added a watch.js file:

const chokidar = require('chokidar');
const _debounce = require('lodash.debounce');
const path = require('node:path');
const { spawn } = require('child_process');

const mixPath = path.resolve(__dirname, '..', 'node_modules/.bin/mix');

const mix = function () {
  const output = spawn(mixPath, ['--no-progress']);
  output.stdout.pipe(process.stdout);
}


chokidar.watch([
   path.resolve(__dirname, '..', 'resources/views/**/*.blade.php'),
   path.resolve(__dirname, '..', 'storage/framework/views/**/*.php'),
   path.resolve(__dirname, '..', 'vendor/**/*.blade.php'),
]).on('all', _debounce(mix, 150));

The relevant dependencies should already be installed with laravel mix & webpack, but just to be sure you might want to:
npm install -D chokidar lodash.debounce

I then added a new item to the scripts in the package.json:
"watch-chokidar": "node ./.scripts/watch.js"

Running that command, watching works as expected once again, and properly re-triggers the mix command any time a file is added, removed, changed, etc.

If you find that it triggers more than once, especially on first run, try increasing the debounce time from 150 to something higher.

Enjoy, hope it helps someone else, and if not it felt good to vent a little bit. 😎

EDIT: Just a week or two later, vite has replaced mix as the preferred/default bundler, and this is no longer a concern - also WOW it's way faster!

I did this config on my previous server, and honestly don't recall exactly what my pagespeed score was before the changes but I want to say it was in the mid 60% range.

There is a simple change that can be made when running Ghost in docker, to make it perform better for the end user – chuck it behind lightify, which will automagically apply a bunch of best practices - concatenation/compression of JS, CSS, etc for you. This lets you go on about the business of blogging without worrying about how to reconfigure things to eek out better pagespeed performance.

Now as you are probably already aware, I run all my various services in docker, proxied behind traefik. It's working fabulously and I have a bit better understanding of it after my recent efforts to move to a new server and reconfigure all the things.

Anyway, I'll keep it short and to the point. Here is basically what my docker-compose file looks lik for this ghost blog:

version: "3"
networks:
  proxy:
    external: true
services:
  ghost:
    image: ghost:3-alpine
    container_name: ghost
    restart: unless-stopped
    domainname: example.com
    expose:
      - "2368"
    networks:
      - proxy
    volumes:
      - ./config/config.production.json:/var/lib/ghost/config.production.json:ro
      - ./data:/var/lib/ghost/content
  lightify:
    image: alash3al/lightify
    entrypoint: ["lightify", "-http", ":8880", "--upstream=http://ghost:2368"]
    restart: unless-stopped
    expose:
      - 8880
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.docker.network=proxy
      # Next four lines handle redirect from www to non-www version of url:
      - traefik.http.middlewares.www-redirect.redirectregex.regex=^https://www.example.com/(.*)
      - traefik.http.middlewares.www-redirect.redirectregex.replacement=https://example/$${1}
      - traefik.http.middlewares.www-redirect.redirectregex.permanent=true
      - traefik.http.routers.lightify.middlewares=www-redirect
      - traefik.http.routers.lightify.entrypoints=https
      - traefik.http.routers.lightify.rule=Host(`example.com`)
      - traefik.http.routers.lightify.tls=true
      - traefik.http.routers.lightify.tls.certresolver=default
      - traefik.http.routers.lightify.tls.domains[0].main=example.com
      - traefik.http.routers.lightify.tls.domains[0].sans=*.example.com

That's basically it. Traefik points the desired host, in this case example.com at lightify, which in turn is a reverse proxy for the ghost container.

Additional nicities in this config, it will redirect from the www to non-www version of the domain.  I also have automatic redirect from http to https setup globally within my traefik's docker-compose file, which I covered in my post yesterday about setting up Mailcow behind traefik, but for brevity here's the relevant labels for that too:

services:
  traefik:
  
    # --- snipping out all the other stuff --
    
    labels:
      # Global Redirect to https
      - traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)
      - traefik.http.routers.http-catchall.entrypoints=http
      - traefik.http.routers.http-catchall.middlewares=redirect-to-https
      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https

That catches incoming http requests, for any host, and redirects them to https instead. Since traefik is grabbing certs for all the things anyway, this works nicely. Caveat– I imagine it would not work so nice if you were dependent on the http challenge type for getting letsencrypt certs, so best to use the dns challenge and/or tls challenge.

There is one small gotcha I've discovered with lightify, for which I have opened an issue on github, if using <link rel="... to load google web fonts, lightify is mangling the urls. The workaround is to instead direcly embed the stylesheets that get loaded.  See the github issue for full details.

All of the information necessary to get mailcow functioning properly behind traefik 2.x already exists on the web, but I found it somewhat challenging to find all the relevant info in one place.

So first off, traefik.  Lets setup a reasonably straightforward traefik instance that'll allow lots of services, and many domains, to run behind it...

Create directory, /opt/traefik and within it create ad directory data where we'll store our traefik.yml and acme.json.  On that note, touch data/acme.json and very importantly chmod 600 data/acme.json ... that last point was a huge head scratcher when suddenly my cert resolver stopped working while I was trying to get wildcard certs setup. If the permissions on that file are less strict, traefik will throw out any resolvers associated with that for storage – but then in logs doesn't mention that, just says "resolver foo does not exist" or whatever. Frustrating.

Ok.. so we've got our /opt/traefik with a data directory that contains an empty acme.json file with appropriate permissions.

Now lets add a traefik.yml file in that data directory too with the following content:

api:
  dashboard: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

tls:
  options:
    default:
      sniStrict: true
      minVersion: VersionTLS12

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false

certificatesResolvers:
  default:
    acme:
      email: you@example.com
      storage: acme.json
      tlsChallenge: {}
      dnsChallenge:
        provider: digitalocean
        delayBeforeCheck: 0
      httpChallenge:
        entryPoint: http

Obviously change the email, and dnschallenge provider should be changed or omit as your situation dictates. I don't know for sure if having all three challenge types in there is correct, I imagine if one fails it'll try the next, etc.. but I don't know for certain. In any case its' working for me, including wildcard cert generation.

Next create the docker-compose.yml file in your /opt/traefik directory with the following contents:

version: '3'

services:
  traefik:
    image: traefik:2.2
    container_name: traefik
    restart: always
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      - DO_AUTH_TOKEN=YOUR_DIGITALOCEAN_TOKEN
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
    labels:
      - traefik.enable=true
      - traefik.http.middlewares.traefik-auth.basicauth.users=USER:HASH
      - traefik.http.routers.traefik-secure.service=api@internal
      - traefik.http.routers.traefik-secure.entrypoints=https
      - traefik.http.routers.traefik-secure.rule=Host(`traefik.example.com`)
      - traefik.http.routers.traefik-secure.middlewares=traefik-auth
      - traefik.http.routers.traefik-secure.tls=true
      - traefik.http.routers.traefik-secure.tls.certResolver=default
      # Omit next two lines to not do wildcard certs:
      - traefik.http.routers.traefik-secure.tls.domains[0].main=example.com
      - traefik.http.routers.traefik-secure.tls.domains[0].sans=*.example.com

      # Global Redirect to https
      - traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)
      - traefik.http.routers.http-catchall.entrypoints=http
      - traefik.http.routers.http-catchall.middlewares=redirect-to-https
      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https

networks:
  proxy:
    external: true

Now since our proxy network is external we'll need to manually create it:

docker network create web

Lets get your basic auth user & password setup too:

htpasswd -nb admin secure_password

Use the hash output from that and replace USER:HASH in the `docker-compose.yml` file with the username you used, and the output hash.

All that's left to do for this part is fire that bad boy up:

docker-compose up -d

Assuming you had docker, docker-compose, etc installed and your dns is setup, you should now be able to access traefik at traefik.yourdomain.com, and if you hit the http version it should automagically redirect to https ... that global redirect to https section of our docker-compose config is pretty magical, it catches any http traffic to any host and redirects to https, automatically.

Mailcow

Follow all the steps for getting mailcow up and running from their documentation.

When editing the mailcow.conf file, be sure to set the following, since traefik will proxy the web connections, and handle cert generation:

HTTP_PORT=8080
HTTP_BIND=127.0.0.1
HTTPS_PORT=8443
HTTPS_BIND=127.0.0.1
SKIP_LETS_ENCRYPT=y

Before you start it though, add a new file, docker-compose.override.yml to the install directory /opt/mailcow_dockerized with the following content:

version: '2.1'

services:
  nginx-mailcow:
    expose:
      - 8080
    labels:
      - traefik.enable=true
      - traefik.http.routers.nginx-mailcow.rule=HostRegexp(`{host:(autodiscover|autoconfig|webmail|mail|email).+}`)
      - traefik.http.routers.nginx-mailcow.entrypoints=https
      - traefik.http.routers.nginx-mailcow.rule=Host(`${MAILCOW_HOSTNAME}`)
      - traefik.http.routers.nginx-mailcow.tls=true
      - traefik.http.routers.nginx-mailcow.tls.certresolver=default
      # Uncomment to use wildcard cert:
      # - traefik.http.routers.nginx-mailcow.tls.domains[0].main=example.com
      # - traefik.http.routers.nginx-mailcow.tls.domains[0].sans=*.example.com
      - traefik.http.routers.nginx-mailcow.service=nginx-mailcow
      - traefik.http.services.nginx-mailcow.loadbalancer.server.port=8080
      - traefik.docker.network=proxy
    networks:
      - proxy


  certdumper:
      image: humenius/traefik-certs-dumper
      network_mode: none
      command: --restart-containers mailcow_postfix-mailcow_1,mailcow_dovecot-mailcow_1,mailcow_nginx-mailcow_1
      volumes:
        - /opt/traefik/data:/traefik:ro
        - /var/run/docker.sock:/var/run/docker.sock:ro
        - ./data/assets/ssl:/output:rw
      environment:
        - DOMAIN=${MAILCOW_HOSTNAME}
        # If using wildcard certs instead of an explicit host cert,
        # use following line instead with just the TLD so certdumper
        # is able to find the cert.
        # - DOMAIN=YourDomain.com

networks:
  proxy:
    external: true

That should be it.  This will get the mailcow web UI running behind traefik using either the FQDN for its cert, or wildcards.. note; if it's the same domain as traefik and traefik already has wildcard certs, you must use the domain name option instead of ${MAILCOW_HOSTNAME} for the domain env setting for cert dumper or it won't be able to find the cert, and if the same domain, you can omit the wildcard cert labels on nginx because that wildcard cert would already be generated by traefik anyway...

Ok, fire that puppy up:

docker-compose up -d

That should get you up and running with mailcow proxied behind traefik. All the mail ports are open directly, but web traffic gets proxied.

If something isn't working, you probably missed a step here, omit a quote or something.. or you have other issues that are related to mailcow – likely dns config or firewall issues. That's all beyond the scope of this post.

Adonis is great.  It, like Laravel, provides a lot of nice functionality that make life easy as a developer. The ORM is pretty pleasant to work with, however it's also very easy to ignore performance and ship something suboptimal without thinking about it.

Which, of course, is exactly what I did.

I'll try to explain without giving away too much inside information about the application I work on full time. In one section of our application we allow users to search for businesses, to generate an audience comprised of many (up to 10,000) locations based on whatever criteria; business name, type of business, etc.

Once a user has made, and narrowed their selections, we generate polygonal data associated with each business' physical location. I'm not going to delve into what else happens as that's well outside the scope of this post.  But suffice it to say, this action results in the creation of many records in the database. Three distinct types of record in fact:

First, there is the audience record itself, this is general metadata for the user's consumption within the application, a name that is shown within lists, etc.

Next, there are two different geometry-based records. One that, again is lots of metadata, the other that contains the actual polygon and so forth.

There are reasons it's split out the way it is... in the simplest of terms it's because the record with the actual polygon can be reused and in a previous iteration of the application that data was duplicated, in several instances many times over.. so normalization was one of the specific design goals when I rearchitected the data structure.

Anyway.. so those records all need to be created, and are all inter-related.. the geometric records belongTo the audience, and belongTo the polygon. So there's a bit of hoop jumping that has to occur to generate things in the correct order.

Previously I was filling the polygon records with the bare minimum content so that I could get an id to populate into the associated geometric records, and then a process further down the line was verifying the lat/lng of the address to see if a better match could be found, then generating the polygon and updating those records – that process is no longer involved, so I can populate polygons immediately, which is nice.

But in either case, what I was winding up with was, essentially, an array of ids to populate into the geometric records as the relation id when I created those, and I was doing so making use of Adonis' createMany() method. Which is specifically intended for that purpose.  Of course most people probably aren't creating upwards of ten thousand model instances at a time... and given that these are individual queries, as is stated in the docs:

The createMany method makes n number of queries instead of doing a bulk insert, where n is the number of rows.

Not really ideal for my use case.  But it's been working alright, just a little slower than ideal.

Enter: Bulk inserts

To be fair, I was already using bulk inserts in the aforementioned bare-minimum insert to get polygon record ids.. a pretty clever query too in fact;

const polygon_id_result = await Database.raw(`
  INSERT INTO polygons (created_at)
    SELECT NOW()
    FROM generate_series(1,?) i
  RETURNING id`, [hits])
const polygon_ids = polygon_id_result.rows.map(r => r.id)

hits here is the number of records.  Postgres has that interesting generate_series function that creates an array.. so the query basically just says "for these n array items, generate empty records containing just a created_at timestamp"

Then I would do my createMany thusly:

await Geometric.createMany(pois.map(poi => {
  return {
    name: poi.name,
    // other fields are not relevant... 
    audience_id: audience.id,
    polygon_id: polygon_ids.shift(),
  }
}))

See, that takes the array of polygon_ids and chucks them in, in order, to the appropriate records in the createMany() method. Works great.. but generates like 10k requests.. Yuck.

Instead now I'm doing bulk inserts for both parts.. making use of lodash's chunk and flatMap though with node 11 or newer flatMap is available natively now...

Basically I build up an array of items as I did before, then chunk that into groups of 500, which is an arbitrary selection, but keeps the overall query sizes reasonable as I'm inserting 5 and 8 columns respectively for each of the two tables' queries.

So now, instead of the bare-minimum bulk insert, I do a bulk insert with actual polygonal data, grab those ids, and feed them in to a second bulk insert into the geometric model's table.  I'm not showing all the fields here, but you get the idea:

// Break up our array of items into chunks of 500
const chunks = _.chunk(pois, 500)

// Iterate through each chunk of 500 items
chunks.map(async (chunk) => {
  const polygons = await Database.raw(`
    INSERT into polygons (geojson, square_meters, metadata, created_at, updated_at)
    VALUES ${Array(chunk.length).fill('(?,?,?,?,?)').join(',')}
    RETURNING id
  `,
  _.flatMap(chunk, (item) => {
    item.geojson,
    item.square_meters,
    JSON.stringify({ address: item.address, point: item.point }),
    created,
    created, // Twice because updated_at == created_at on creation.
  }))

  // Extract record ids from the query result
  const polygon_ids = polygons.rows.map(r => r.id)

  // Time to replace that poorly performing createMany() with some bulk 
  // insertiony goodness...
  await Database.raw(`
    INSERT INTO geometrics (name, audience_id, polygon_id)
    VALUES ${Array(chunk.length).fill('(?,?,?)').join(',')}
  `, _.flatMap(chunk, (item) => {
    return [
      item.name,
      audience.id,
      polygon_ids.shift(),
    ]
  }))
})

Ok.. that's a little involved, lets walk through it.  First we split the full array of items into chunks of 500, then iterate through each chunk and insert polygons, returning the created ids.. then take those created ids, and insert values into the DB for the geometric models associating the appropriate polygon id with each.

The interesting bits here, to me anyway, are the Array().fill() which takes the number of items in the current chunk, and creates an array with that many instances of the value passed into fill(), in this case, the placeholders for each record being inserted. That array is then join()ed with commas to generate the string of placeholders for each row to be inserted.

The resulting queries end up looking something like this:

INSERT INTO foobar (foo, bar, baz) 
VALUES (?,?,?),
       (?,?,?),
       (?,?,?),
       (?,?,?),
       (?,?,?)

Extra linefeeds added for clarity, of course

Then the flatMap takes the array of items, and for each returns an array of the data for each of the relevant placeholders – which then gets flatenned (hence the name), so that it effectively goes from something like this:

[
  { color: 'blue', flavor: 'raspberry', id: 1 },
  { color: 'red', flavor: 'strawberry', id: 3 },
  { color: 'green', flavor: 'apple', id: 7 },
]

To this:

['blue','raspberry',1,'red','strawberry',3,'green','apple',7]

Which is the format that has to be passed in as replacements for the placeholders when running those queries.

At the end of the day, this change goes from insertion/creation of 1 audience, 10,000 polygon records, and 10,000 geometric records taking a full minute or longer, to completing in about 6 seconds.

Additionally, it's far fewer queries to the database (41 if my math is correct, vs 10,002 or so previously), which means far fewer network calls that could potentially fail, and far less likely that the request will simply timeout before completing at all.

41 still sounds like a lot of course, and it is.. but it's not even the same sport, let alone in the same ballpark as 10k queries.

I anticipate utilizing this pattern more often, and while working out how to manage the population of the placeholders, and mapping the data for the replacements presented a bit of a challenge until I thought to use Array fill and flatMap, it's one of the more interesting solutions I've found lately.

To make a long story short; if you're inserting more than maybe a dozen or so records, you almost certainly do not want to use createMany() ... whip yourself up a nice performant bulk insertion instead, and avoid the problems poor performance cause before you create them in the first place.

Given the similarity between Adonis' Lucid ORM and Laravel's Eloquent, I suspect virtually the exact same solution would also apply to Laravel.. with the obvious PHP syntax changes of course.