Okay, so DNS doesn’t really work like that. The client will contact the first DNS server and try for resolution (namely the Windows server) for client.b.domain.com. The windows server, not knowing who *.b.domain.com is, will return that there’s no entry by that name, the client will accept this as truth (aka an authoritative response) and stop. The second DNS resolver will not even be given the query.

The way to do this is to have a master for the zone. It doesn’t matter if this is on the windows side or linux side, it just needs to exist. the master zone (*.domain.com) then dictates NS records for the two subdomains, eg:

a.domain.com NS (ip of windows server) b.domain.com NS (ip of linux server)

The window DNS then has an authoritative zone for a.domain.com, which contains all the records relevant to the operation of that domain. linux has the same setup, but for b.domain.com, which has all the records for the operation of that domain.

What happens then, is the client queries the DNS primary server for client.b.domain.com, gets back a response that basically says the princess is in another castle, look up the NS for that subdomain (b.domain.com), and queries the provider for b.domain.com (the linux server) and gets the correct response… at least, that’s how it should work. Clients vary from OS to OS, so the DNS server may handle this request instead, forwarding the request (though this may or may not happen, depending on configuration).

IMO, the best way to do this is to have a small DNS-only system which acts as a relay/caching server that everyone points to, it has two forwarding zones, one for a.domain.com and one for b.domain.com, with the records set to forward requests to either the windows server IP or the linux server IP based on request. There should still be a master zone for domain.com which has the NS records, which bluntly can be the relay and caching server. What I like to do is have a small linux system for this, which has a global resolver set for the fall-back DNS (I usually use Google at 8.8.8.8/8.8.4.4), with domain.com defined, and two NS records there for a.domain.com and b.domain.com, pointing to each server, then have forward zones for a and b respectively. I would duplicate this, and provide two DNS resolvers as relay/cache, and set the windows and linux master servers (for a.domain.com and b.domain.com respectively) as non-recursive.

Super fast: recursive resolvers basically allow you to resolve non-authoritative names. An authoritative response is when the DNS server is the root server for that domain or subdomain; aka the NS for that domain, aka, it physically holds the records for the domain, and doesn’t need to ask another server for that information.

A bit of a hacky way around this, without adding more systems, is to add a forwarding zone to the windows server for b.domain.com that points all queries to the linux server, and a forwarding zone on the linux server for a.domain.com that points to the windows server. You can keep your current primary/secondary DNS entries, and both servers will be able to resolve both sets of names. The DNS servers will simply forward request of the opposing domain to the respective authoritative server.

Doing this “correctly” needs about five DNS servers, one for the “master zone”/root of domain.com, one for each of the subdomains (both a and b) and two for caching/relay, though, each of those should be redundant, so two windows and two linux authoritative zones for the subdomains, and two root servers - at least - to get everything to be “proper”.

DNS is fascinating and complex; I see what you were trying to do with your current setup, that the clients will try the primary, and fail, then try the secondary, and get a reply, unfortunately, that’s not how DNS works. It only fails over to the secondary when the primary doesn’t respond. Having redundant DNS is a really good idea regardless. What I do, is I have a pair of Raspberry Pi units on my network (currently both model 3B’s), which have BIND installed, and forwarding zones for my internal domains plus a global forwarder for anything outside that, pointed to google, as previously mentioned. They act as relay/caching, and anytime I want to put something together, either to resolve DNS names only available over a VPN or for a new subdomain or internal domain, I just add it to those BIND servers, pointed at the authoritative name server (NS) for that domain name and it works pretty well. You don’t have to do the same, nor do you need to do something similar. There’s a lot of good ways to configure this that will work. There are also some really terrible ways to set this up that will give you nothing but problems. I set mine up the way I do because those r.pi units are PoE powered, so as soon as the network is online, they come up and work, keeping my partner happy, while I can fiddle around with DNS forwarders to get my lab stuff working. When the lab is down, the internet still works, so I don’t get flack for screwing with the internet again. My main concern is to have redundant DNS that works whether my lab is powered on and working correctly, or not. So if I lose my lab, or lose a DNS server, the internet still works and I don’t get yelled at again.

Let me know if you have any questions or follow ups, I’m happy to share the knowledge I have.

Good luck!