The variables of an initiative system
The most common division made between initiative systems is in the granularity: individual initiative versus side initiative.
In individual initiative (the default in 5th edition D&D), the initiative order is written down, with each character acting on their own initiative count. Sometimes in battles with several types of monster, the monster types are grouped to act on the same count, but the key part is that each PC gets their own count, and their individual traits (Dexterity score, feats) affect that count. The benefit is that players get to feel the rewards of investing in those aspects of their character. The drawback of individual initiative is the need to constantly consult the initiative track, and the process of writing it down takes long enough to take us out of the exciting "combat is happening" moment.
In side initiative, the players and monsters make a contested roll to see which faction gets the first turn. This is then repeated (either in the same order, or re-rolling at the top of each round). Players are free to choose in which order they want to act during their side's turn. The benefit of side initiative is that it's fast, simple, and there is no need to write anything down, so it doesn't take us out of the moment. The drawback is more apparent in systems where action economy is important; if all the monsters focus-firing a single PC can down them in a single turn, fights can become even more swingy than they already are, with the first turn deciding it all.
But this "side/individual axis" is only one variable of initiative systems. As an example of another variable, you could disregard rolling entirely, instead using the characters' passive initiative modifiers to determine their order (the 5e DMG presents this variant as "initiative score). This speeds things up, but only slightly, as the initiative order still needs to be written down. It also makes it rather static for the players, who end up acting in the same order relative to each other every time. It probably depends on your players whether they would be bummed out by that, or enjoy using it to discover trademark strategies.
This presents a second axis along which to customize initiative systems: the contested/passive axis. You can even introduce a hybrid of the two, in which only one side rolls, using the other side's passive score as a target number, but I see no particular benefits in this and only mention it as a curio.
Setting-specific initiative
To recap, the default 5e initiative system is a "individual + contested" initiative, in which everyone rolls, contested by everyone else, and the DMG presents an "individual + passive" variant. Many old-school games such as Moldvay Basic use a "side + contested" system as the default. But to really add flavour to our initiative system, we can create a "side + passive" system, and make the deciding factor something other than Dexterity score.
Veins of the Earth has "Lamps are Initiative", which has only the people with the brightest lights roll initiative - but if you decide that those with the brightest lights always go first, no rolling, you've got something interesting going. To give another example, consider what I'm calling "guerrilla initiative". Under this system, instead of rolling initiative, the side with the smaller numbers always has initiative. The players storm the lair of a solitary dragon? The dragon reacts first. The lair has a dozen kobolds in it? The players go first. The rule communicates something about the world: it really emphasizes the stealth in small numbers and the overhead of organizing as a larger group. You can come up with an infinite number of ways to decide which group has initiative in each setting.
Putting that aside, the system that I feel like combines the most strengths with the fewest drawbacks is something that, strangely, I haven't seen in any books. I call it split-side initiative.
Presenting split-side initiative
Like Moldvay and 5e, this is a contested roll, but it sits between side initiative and individual initiative.
The DM rolls a single initiative check for the monsters's side. The players then all roll initiative individually, using the monsters' roll as a target number. Players who roll equal or greater than the monsters' number are in group A. Those who do not are in group B. The turn order for a round then goes: the players in A, the monsters, the players in B. The player side is "split" in two. Each group of players acts as they would in side initiative.
This system is very simple, almost as simple as the old side initiative. But it also allows player characters to differentiate (rewarding high Dexterity scores), and reduces the impact of the "everyone focus-firing down a single character" problem. It's equally usable in old-school games, and games like 5e with a variety of feats and class features that interact with initiative rolls.
What do you think? Do let me know if this variant has been included in any books. It must have, seeing as there's no such thing as an original idea, right?
