u/ClearlyTheWorstTech

140 post karma · 2,049 comment karma · joined Jan 2021

ClearlyTheWorstTech

As someone who has written tons of exchange rules in tenants and actually looked up the hierarchy of rules it goes like this:

Forwarding > exchange mail routing > mailbox rules

The reason for this is a little ambiguous, but forwarding implies that any mail hitting a mailbox is forwarded before it is processed by exchange routing. It has to do with when email headers are processed. You sound like you're pretty green when it comes to exchange / 365. Leveraging Microsoft Graph, powershell, and an Ai chat box will allow you to create lists of accounts with email forwarding enabled or any other attributes you want to look for and then forcefully remove them.

ClearlyTheWorstTech

Oh! Please, take this into consideration when choosing your remote access tool. Do you get support from outsider Vendors and are you familiar with their remote access clients? Tools like Datto RMM, VSA10, and Splashtop (not QS) will overwrite other remote access clients as the tool can only function while attached to one monitoring instance. Meaning if you both run the same tool. On installation for support you will lose access. We recently started having this issue for some of our veterinary customers because their parent company IT support in another state were using Datto RMM. We kept stealing clients from one another for over a month before they gave us a call.

Also, with a company as large as yours you have to be using a ticketing platform, documentation platform, and a keyring tool, right? If not, you should have one of each for business continuity and to keep above the HBAB line of an unrecoverable situation.

So, Apple iOS devices should be managed by MS Intune, Jamf, Mosyle, or Addigy. Mobile Device Managers make the setup of personal iCloud accounts on endpoint phones an impossibility. If you currently experience issues with phones, purchased by the company, being tied to personal accounts and being unable to redistribute the devices because Apple won't release the device from the "find my iPhone" service after an employee termination. MDMs allow you to get around that and be able to nuke a phone from the cloud if lost or stolen.

When I departed my previous position, we were utilizing Splashtop Pro in addition to ScreenConnect, ConnectWise Manage (locally hosted), Webroot, and ITSupport247. Before this we were running LogMeIn with the other listed tools. ITSupport247 switched from LogMeIn to ScreenConnect after acquisition and allowed existing customers to deploy ScreenConnect for Theta-testing in their environment. ScreenConnect almost never disconnected due to low bandwidth and it never stopped rendering frames. If a connection was bad it would stutter and show your cursor being inches behind your own cursor. When that happened all I needed to do was drop the quality of the session into the Low setting with no color, but doing so allowed us to work with clients running on 2-5 Mbps connections.

We had purchased Splashtop to run beside LogMeIn because the LogMeIn client would frequently lock us out if a client changed passwords. Imo, LogMeIn was developed to run in a domain environment and nothing else.

ITSupport247, BTW, is Connectwise's supplemental NOC, RMM portal, BCDR Management, and Monitoring platform for customers willing to pay for the service. They're based in India, but they have managed to retain most of their Technicians without a significant amount of turnover and all of them are up-to-date on their own implemented software and on their Microsoft Server Administrator Certifications. We frequently worked with the same people on their team due to how their Shifts aligned with our Time Zone over 2 years. Previous company owner made a point to learn the technicians names and they really tried to work as a team with us. The Splashtop licensing was separate.

Splashtop did do okay with low-quality connections after dropping quality like with ScreenConnect. I frequently found ScreenConnect performing better though.

I would argue that any true RMM platform with a dashboard, management agent, and separate remote access agent is still better than just ScreenConnect, Splashtop, TeamViewer, LogMeIn, etc. with their standard remote access portals. What I mean by that? The ITSupport247 team has a separate agent running on each computer for running remote commands to repair the remote desktop agent, to automate installations, and run cmd/powershell commands. Both VSA and Datto RMM are built this way. I personally think any organization is doing themselves a disservice by not leveraging a tool that can push automation scripts to endpoints reliably.

The worst issues we have had with either VSA and Datto RMM has been caused by either Microsoft Updates without rebooting or the VSA / Datto RMM agent updating and failing to load or having a partial installation. VSA 9 never addressed this issue and Datto RMM finally has an entry in their knowledgebase for which firewall rules must be enabled on remote endpoints to allow remote deployment/redeployment of the agent from a Network Node PC or server with local or domain administrator credentials. I believe VSA10 also has this fixed, but I don't know for sure as I haven't played with it as much. On Domain networks you can leverage the auto discovery tool by setting up the domain controller and setting it as a network node with scanning or adding a GPO to auto-install the Datto RMM agent. The GPO setting you need to add? Just push the "Job" from the Automation tab as a pre-built "Component" found in the Component Library maintained by Datto.

ClearlyTheWorstTech

No worries, mate! Let me know if you have any questions.

I was curious though, could you share the type of environment your company is licensing remote management for? MSP with multiple companies? One company with one location? Or one company with remote locations? It would help someone like me to recommend one specific solution over others if you are interested.

ClearlyTheWorstTech

So, I have had the opportunity to use TeamViewer, LogMeIn, GoToMeeting, Splashtop, ScreenConnect, DWService, Kaseya VSA 9, Join.Me, VNC, Microsoft Quick Assist, and Datto RMM.

I personally feel like no one solution is perfect for everyone. It depends on your needs and your current stack. If you already have things like documentation, keychain management, and endpoint security systems? Then it's better to get something that integrates with those existing systems. If the only system that does integrate has been declared as unusable by your department? Oof.

TeamViewer's new licensing model is trash. It's not very competitive when compared to other pricing models out there. TeamViewer is however the most versatile in endpoint deployment and ease of configuration with the quick support platform. If you are just managing endpoints? It's not worth it.

LogMeIn is behind the times and requires far too much credential management for the technicians to allow remote access. The backend tools are also behind on capability in my opinion and are not a selling feature of the platform. Ever since Gotomeeting was picked up by LogMeIn they're equivalent. Join.me was the best product that LogMeIn stifled the development on to make their remote whiteboard meeting tool fail against Zoom.

VNC is great for an Air-gapped system that will never see the light of ISP Fiber Optic. Otherwise it's a free exploit for attackers.

Splashtop has long been my favorite solution for mixed environment. They really have their Apple Mac OS client configured well. I like splashtop because it's a very good remote access client and it doesn't try to be much more than that. I like their quick support option and their dedicated client being able to quickly and easily switch between either option.

Microsoft Quick Assist is by far the easiest and best option to help someone get to the correct web page to download another remote assistance solution. The best/worst caveat being the limitations of the software being in user mode.

DWService is what I use for my home computers and my homelab. It's still free for the time being, but they have a very modest price model for what is honestly a better resource than more than half of the other products I have used. It has a backend cmd / cli access. It functions on practically every platform. I can access computers from my web browser or from my phone.

My favorite resource for remote access and working on remote computers is definitely ScreenConnect. It appears at first glance to be a splashtop clone, but you can pin your top toolbar to a dedicated sidebar. You can switch from their desktop to the backend which has been completely outfitted with shortcuts and a GUI taskbar alongside the CMD and Powershell windows. It is an insanely powerful tool and it grants you access to nearly all of the windows environment CPL and MSC tools from the background without interrupting end users. I miss using it. I don't know what it integrates with as my experience with it was from when there was only CMD and Powershell in the background instance. This was the first year or so after ConnectWise either acquired ScreenConnect or purchased licensing for it.

Kaseya's VSA solution is a sprawling broken mess. I have a resentful disposition towards VSA 9. It's still supported despite losing several built-in modules that were accessible from the side-bar navigation. Those modules include a ticketing system, antivirus, and more. The "procedures" in VSA 9 allow for Automation scripts, but they are the furthest thing from user friendly. VSA 9 has a drag and drop script building utility for the Procedure manager that needs practically every special character escaped to function properly. The main reason VSA 9 is still supported is due to the fact that you can run your own instance of the VSA Server locally and in air-gapped systems. VSA 10 is a whole new platform with many of the features from VSA 9, but it comes with scripting workflows for automated solutions that can run when defined monitors are triggered. These workflows also are able to be written without escaping tons of characters that are constantly used in scripting on windows. VSA 10 has a lot of integration available, but it still uses the same remote access client base used by VSA 9. Most of the Kaseya products can integrate with VSA 10 and it's a viable option. Cost per endpoint ranges from $3.50 to $5 or something? I don't remember exactly. I just remember that it comes with several additional software bundled into Kaseya K365. Their sale pricing is sometimes amazing and if you get any products on sale it is something that their sales team will push you to lose. They will fight tooth and nail to get you back up to standard pricing.

Another Kaseya product is Datto RMM. I won't lie, it's our company's current solution. It's very versatile as it comes with splashtop, tunneled VNC, their own remote "Web Access" that can be used by any browser on any device as it runs on HTML5, the Agent Browser client with a plethora of tools, RDP, automation tools that do not require escaping characters, and more. The best part about Datto RMM is the integration though. It Glue is a companion software for Datto RMM that adds company passwords, organized to match your RMM sites and be directly accessible via the web remote sidebar. It grants you the ability to input saved and shared credentials as keystrokes directly from the sidebar. The management interface for endpoints comes with a dashboard that integrates with their Avira-based EDR platform. It even has the ability to integrate with Datto Autotask ticketing system in a way that reminds you to enter notes from the remote session when you close the Agent Browser. It has the same pricing as the VSA solution, but so much of the Datto RMM Solution is geared towards integrating the tools in the stack in plausible and helpful ways. Even if it isn't my favorite. I find myself losing the argument to switch to another solution based on the usability, versatility, and just flat out useful features. Making endpoint computers into network nodes allows you to leverage the Agent Browser to SSH, Telnet, and access HTTP/S sessions remotely on the attached network. Just be sure to set your default browser to chrome before you begin an HTTP/S session as other browsers do not tend to route traffic successfully.

ClearlyTheWorstTech

The only thing that has changed is the UI.

ClearlyTheWorstTech

As someone who has worked with Worldship on a domain and had to learn the infrastructure on the fly. The worldship database is running on another computer and the endpoints in the warehouse are running as access.

Worldship has been developed and shipped as a "Standalone database server with shipping front-end that can run on any Windows computer".

This is fantastic for most small businesses. Set it up on a dedicated cheap pc with a printer and move on.

UPS decided they needed to make it scalable, but they don't have a single IT security expert in their development or support ranks. If you follow the support guidelines? You run it as admin in every instance. If you have the domain handling access to the database? It must be a domain user to get access to the share / server the database resides on. If anyone contacted UPS support for setup? They request admin access for every step of installation. There's about 4 or 5 prompts during installation as well. They can't comprehend a system without root access and they don't develop for it.

The best solution imo is to run either without admin as the base domain user, set permissions on all of the local worldship folders for domain users, and push updates for worldship with set periods of downtime. If you want anyone on the floor to run updates (common in smaller companies) then use the lusrmgr.cpl to grant specific domain users admin rights on the local PC for running updates. On the server computer it is always recommended to grant user permissions on all of the associated folders, databases and files for the domain users who will be running worldship remotely.

My guess is that this warehouse is operating 12-24 hr days and they ran out their kerberos authentication for the one or two Worldship PCs that have been up and running for months. If you force an automated log-out or restart before the kerberos timeout this shouldn't happen again.

ClearlyTheWorstTech

In moments like this. Once I feel like I understand the position, I feel like I need to talk to HR or whomever interviewed / hired me and request they change my job title to suit the position. IT Liason and Data Analytics, IT Coordinator, or something. It seems like there are too few people who can adequately explain IT Systems and Concepts to normies. I frequently come across those who bullshit C-suite by using buzzwords or abbreviations C-suite won't know to bolster every task they undertake. It's frustrating as hell to watch them in real-time exaggerate a 2-hr task like they held up one end of a banquet table during the CEO's Invite-only investor's meeting and saved the company.

ClearlyTheWorstTech

This my guy. It's even more insulting to register that they copypasta chatgpt without even changing the font to hide it. They just blatantly post what the chat bot said as fact and nitpick anything you send them.

ClearlyTheWorstTech

I am not familiar with the tiny PXE platform for use as a DHCP server, but the BMC port might be sending bootp requests instead of just DHCP and it is rejecting any default bootp/tftp parameters. Also, if you are using different switches than production? They might be configured to reduce broadcast and multicast traffic on specific ports. That would result in a shared BMC port to be flagged or limited in the number of DHCP broadcast requests that could be sent by the one port. It's usually recommended to set BMC to static and then exclude the BMC addresses because of this as well.

ClearlyTheWorstTech

Very accurate for us poor small-house MSP groups. We run networks for over 20 companies and have less than 10 employees. Being able to drop a workload to fix something major for another company before going back to the other work that piled up is always a struggle. My MSP works hard to guarantee network hardware functionality and recommends all the best options we can. It's the Administration / C-suite that usually decide that our suggestions aren't worth the cost until something blows up.

ClearlyTheWorstTech
  1. Get close with Front Desk and Finance. They are the most important people in your office besides the C-suite/President. They know everyone and sign your check. Keep them happy and you will always be notified of deliveries, free food, and who to be wary of.

  2. Automate your winget updates.

  3. Finish the windows 11 upgrade because lord knows the last guy didn't. Replace old equipment that doesn't upgrade to 11. Don't fold on this. Having a backbone in a new position is essential.

  4. Create a network map and topology. Work on getting access to ALL of your network equipment. Find every small unmanaged switch both logically and physically

  5. Repurpose a server as an XCP-NG platform for virtualization. Or proxmox if you can.

  6. Look at getting remote desktop solutions. Splashtop, ScreenConnect, DWService, etc. You will cherish not having to wear out the soles of your shoes running between frantic calls.

  7. Personally can't recommend IT Glue enough. Network Glue (sub product) can help with mapping your network.

ClearlyTheWorstTech

Through Kaseya. We are already on their platform, but from what I understand they sell it independently and through Kaseya/Datto.

I am not an account manager or anything and I don't really recommend getting under their "umbrella", but they have a couple of good products. The best grouping is the k365 endpoint and k365 user. Most of which is advantageous because it has been designed to function together. Datto RMM to manage endpoints and equipment. Autotask for ticket management. IT Glue for centralized password management and documentation. Those synchronize together alongside the Datto Endpoint backup and Datto EDR. The IT Glue functionality is immediately accessible from a remote access tab in your browser and it works on mobile devices to access computers.

No matter how good their products are working together, though Kaseya is working against you. Other products that were great might start losing functionality as they rip portions of the product off to use in another product they're trying to sell. They will frequently try to strong-arm you into contract up-sells and sometimes the sales manager will straight up change the contract details before they give you the paperwork to sign. They are for profit. Not for MSPs.

ClearlyTheWorstTech

We use Bullphish. It's an MSP tool with integration and education for both platforms. They include templates and attack profiles that can be specified per organization in both M365 and G-suite/Workspace

ClearlyTheWorstTech

Other helpful things to type into your start menu to gather information from your computer. I will not provide anything that shouldn't already be present on your computer without a warning:

Winver

Gets your windows version information in a readable format.

Msinfo32

Displays hardware information for your computer like your motherboard model, manufacturer, your CPU, total RAM, and more.

TPM

This is the Windows tool to enable TPM if it's already enabled in the BIOS and you need windows to configure the TPM so that windows update will work again.

https://support.microsoft.com/en-us/windows/experience/compatibility/check-if-a-device-meets-windows-11-system-requirements-after-changing-device-hardware

The above link is for downloading the PC Health Check app also known as the Windows 11 readiness tool that will assess your computer and explain why you can't install windows 11.

ClearlyTheWorstTech

So, windows 11 requires that you have a finicky piece of hardware called a TPM 2.0 (Trusted Platform Module) or better loaded on your computer through an expansion card, the processor (CPU) itself, or the motherboard. Several custom-built computers don't have TPM enabled by default even if they are able to provide the same service Windows 11 is looking for. Several manufacturers name it different things as well. Like Embedded Security Device. Intel PTT. AMD fTPM. Etc.

This post details some of the various settings in your Motherboard BIOS that changes depending on the Brand of motherboard. https://www.reddit.com/r/MSI_Gaming/s/Lj7kpN6KnV

If your processor is not supported, you can check at places like this:

For intel

https://learn.microsoft.com/en-us/windows-hardware/design/minimum/supported/windows-11-supported-intel-processors

For AMD

https://learn.microsoft.com/en-us/windows-hardware/design/minimum/supported/windows-11-supported-amd-processors

ClearlyTheWorstTech

10/10 agree.

Honestly tho? If they have 4 techs and 100 endpoints? That seems like an excessive workforce.

If the plan is expansion? Sure, but they should probably go wholehorse and get the entire itsupport247 or connectwise msp platform to get RMM, security, afterhours support, ticketing, etc. My last job used the old connectwise manage company-hosted platform and it was excellent for ticket/email management with sensible merging rules with timestamp retention. It looks like the new online platform took a sharp turn in a better direction though.

ClearlyTheWorstTech

This is one of the most sound suggestions besides asking the obvious question of "which idiot bought switches that require meticulous configuration?"

ClearlyTheWorstTech

My three most-used commands:

Rename-computer [-new name] $newpcname

Get-printer | select-object -expandproperty name,driver name,portname

Iwr -useb https://christitus.com/win | iex

That last one is one I exploit to repair winget and load chocolatey during a site visit. It's easier to type that line, install 7-zip (or another equally lightweight app), then install whatever else I need with chocolatey. I also use it currently to clean up windows 11. Shout out to Chris for an amazing tool.

Get-printer will allow you to get started into the printer management powershell realm. Where you can start scripted printer deployments in your non-AD environments. Super useful when paired with RMM software.

A cmd tool I Also use frequently?

Netsh wlan export profile key=clear folder="d:\wlan_export"

for %%a in ("%~dp0wlan_export*.xml") do (netsh wlan add profile filename="%%a" user=all)

OH MAN! HOW COULD I FORGET?!

$somecommand | more

Useful across all platforms (ms/apple/Linux). Allows you to scroll with enter or spacebar and reduces your command output to the size of your cmd/powershell/terminal/ssh window. Also, you can exit further output with ctrl+c

ClearlyTheWorstTech

Wait are you serious? Hang on. I gotta verify version implementation for this shorthand.

EDIT: I am becoming one with the rock I live under. 11 years ago there was a thread about using the foreach shorthand.

ClearlyTheWorstTech

I work as a contractor for an MSP. Anyone talking the way you are, right now? They should just hire an MSP and pass the requirements on to the MSP 7/10 times the monthly msp fee goes towards real security software (edr, mdr, firewall, network management, etc). Plus, passing PCI to the MSP allows them to keep your environment compliant in more simple ways if they're competent. I receive about 10 PCI tickets a year and we are already ahead of the curve with our lower tier of monitoring/protection ($13/unit/month). Any MSP worth their salt will protect the client first and get signed waivers for passing up on security. I am a little confused by your statement though, if PCI is just a racket, why were they ahead of the insurance companies by 24-months on security compliance? It still blows me away when I hear how much these companies are willing to throw away on insurance instead of security.

ClearlyTheWorstTech

There are literally so many good/better ways to handle this. Media server? Just allow connections from your local company subnets. You could literally just cast videos to your network with VLC. Embed the video on a website. Upload it to a private YouTube page.

The only use-cases I can imagine for this involves an eye-patch and some rum.

ClearlyTheWorstTech

Sounds like the perfect time to make some half-ass repairs so then the sharks and the bottom feeders can get a new kind of appetite.

ClearlyTheWorstTech

Honestly a little tired and the dyslexia is more of an issue when I'm tired. That's on me.

ClearlyTheWorstTech

So, others already said it, but is this locally at your dad's company that you complete these deployments? Or is this on-site for the client?

You should be able to do a scripted PXE boot. IIRC you can run windows server without a license as a PXE server or you can run a kms activation for windows server and then run PXE on it.

For any of your Datto clients I recommend scripting using the Datto Components and then mapping the component to a job to run after an Initial Audit. This is the first step that takes place after Datto RMM is installed. You can set up the initial audit jobs to apply to specific companies in Datto RMM. If you have the free-time to script these? 10/10 idea. I currently don't have enough time to throw away on that configuration.

If you can edit your unattended to include a runonce registry entry after the installation is finished? Then you can cause the script to fire after reboot. Your unattend file should be completing your windows OOBE with a default account. If you aren't already, build your unattend and script files to be specified to the "most default" setup for the company it is assigned to. Or build multiple dependent on the type of units required by staff. (designer vs accountant).

Currently, we utilize the default setup, skip the unattend and reinstall rigmarole to instead just script common uninstall, company specific 3rd party apps, Wi-Fi networks, printers, and domain if applicable. We keep point-to-point vpn available to map to their local domain remotely. Just have to remember to connect it before we start setup. We also usually do profile migrations also, but that's the most time-consuming part. I wish I could make that part take less time.

ClearlyTheWorstTech

I think he means endpoints on Datto RMM