This is not anything new and has nothing to do with Powershell.
Your position of:
'I can create a file with a colon : in the name which should not be possible in Windows.'
... is not valid.
This has been possible in WIndows for decades. It's called ADS (Alternate Data Streams) and this is not a folder, but simply extra data (text. pictures, etc.) embedded/appended/hidden in the data file. This is a well-documented thing. ADS will never increase the original file size, by design.
This is also a security thing called Steganography, and yes, when you use such techniques, you need to know exactly what the stream name is.
Just do a search for 'Windows Alternate Data Streams' or 'windows alternate data streams view' and you'd get lots of hits on what it is, why it exists, how to do it and how to read them.
You can just use the Microsoft SysInternals tool called streams.exe to see them as well.
https://www.bleepingcomputer.com/tutorials/windows-alternate-data-streams
You could also just search for PowerShell read 'alternate data streams' to see how to read them.
Lastly, as for this...
'Remove-Item -Path .\test:test.txt works, but the pseudo folder remains.'
... that too is not valid. When you delete the main file, anything associated with it is removed, since it only exists because the main file exists.
There is a difference between removing the ADS and deleting the whole file.
$TargetPath = 'D:\Test\ADSStuff'
Set-Content -Path "$TargetPath\test:test.txt" -Value "colon filename test"
Get-ChildItem -Path $TargetPath -Recurse
Get-Content -Path "$TargetPath\test:test.txt"
# colon filename test
Remove-Item "$TargetPath\test" -Force
Get-ChildItem -Path $TargetPath -Recurse
Get-Content -Path "$TargetPath\test:test.txt"
<#
Get-Content : Cannot find path 'D:\Test\ADSStuff\test:test.txt' because it does not exist.
At line:1 char:1
+ Get-Content -Path "$TargetPath\test:test.txt"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (D:\Test\ADSStuff\test:test.txt:String) [Get-Content], ItemNotFoundException
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetContentCommand
#>